为什么即使堆等大小稳定，Sun JVM也会继续消耗更多的RSS内存？ [英] Why does the Sun JVM continue to consume ever more RSS memory even when the heap, etc sizes are stable?

问题描述

在过去的一年里，我对应用程序的Java堆使用量进行了大幅改进 - 减少了66％。为了实现这一目标，我一直在通过SNMP监控各种指标，例如Java堆大小，cpu，Java非堆等。

Over the past year I've made huge improvements in my application's Java heap usage--a solid 66% reduction. In pursuit of that, I've been monitoring various metrics, such as Java heap size, cpu, Java non-heap, etc. via SNMP.

最近，我一直在监视JVM有多少实内存（RSS，驻留集）并且有点惊讶。 JVM消耗的实际内存似乎完全独立于我的应用程序堆大小，非堆，eden空间，线程数等。

Recently, I've been monitoring how much real memory (RSS, resident set) by the JVM and am somewhat surprised. The real memory consumed by the JVM seems totally independent of my applications heap size, non-heap, eden space, thread count, etc.

测量的堆大小by Java SNMP
Java Heap Used Graph http：/ /lanai.dietpizza.ch/images/jvm-heap-used.png

以KB为单位的实内存。 （例如：1 MB KB = 1 GB）
Java Heap Used Graph http://lanai.dietpizza.ch/images/jvm-rss.png

（堆图中的三个凹陷）对应于应用程序更新/重新启动。）

这对我来说是个问题，因为JVM正在消耗的所有额外内存都是窃取内存，可能是OS用于文件缓存。事实上，一旦RSS值达到~2.5-3GB，我开始看到应用程序的响应时间更慢，CPU利用率更高，主要是IO等待。因为某些点对交换分区的分页启动。这都是非常不受欢迎的。

This is a problem for me because all that extra memory the JVM is consuming is 'stealing' memory that could be used by the OS for file caching. In fact, once the RSS value reaches ~2.5-3GB, I start to see slower response times and higher CPU utilization from my application, mostly do to IO wait. As some point paging to the swap partition kicks in. This is all very undesirable.

所以，我的问题：

• 为什么会这样？发生了什么幕后


• 我该怎么做才能保持JVM的实际内存消耗？

• Why is this happening? What is going on "under the hood"?
• What can I do to keep the JVM's real memory consumption in check?

血腥细节：

• RHEL4 64位（Linux - 2.6.9-78.0.5.ELsmp＃1 SMP Wed Sep 24 ... 2008 x86_64 ... GNU / Linux）


• Java 6（ build 1.6.0_07-b06）


• Tomcat 6


• 应用程序（按需HTTP视频流）
  
  
  
  
  • 高I / O通过java.nio FileChannels
  
  
  • 数百到数千个线程
  
  
  • 数据库使用率低
  li>
  
  • Spring，Hibernate
  
  
  • RHEL4 64-bit (Linux - 2.6.9-78.0.5.ELsmp #1 SMP Wed Sep 24 ... 2008 x86_64 ... GNU/Linux)
  • Java 6 (build 1.6.0_07-b06)
  • Tomcat 6
  • Application (on-demand HTTP video streaming)
    • High I/O via java.nio FileChannels
    • Hundreds to low thousands of threads
    • Low database use
    • Spring, Hibernate

    相关的JVM参数：

    -Xms128m  
    -Xmx640m  
    -XX:+UseConcMarkSweepGC  
    -XX:+AlwaysActAsServerClassMachine  
    -XX:+CMSIncrementalMode    
    
    -XX:+PrintGCDetails 
    -XX:+PrintGCTimeStamps  
    -XX:+PrintGCApplicationStoppedTime  
    -XX:+CMSLoopWarn  
    -XX:+HeapDumpOnOutOfMemoryError 
    

    我如何衡量RSS：

    ps x -o command,rss | grep java | grep latest | cut -b 17-
    

    这会进入一个文本文件并被我的监控读入RRD数据库系统定期。请注意，ps输出Kilo字节。

    This goes into a text file and is read into an RRD database my the monitoring system on regular intervals. Note that ps outputs Kilo Bytes.

    最后它是 ATorras 的答案最终证明是正确的， kdgregory 谁使用 pmap 引导我到正确的诊断路径。 （对他们的答案进行投票！）以下是发生的事情：

    While in the end it was ATorras's answer that proved ultimately correct, it kdgregory who guided me to the correct diagnostics path with the use of pmap. (Go vote up both their answers!) Here is what was happening:

    我确切知道的事情：

    Things I know for sure:

    
    
    1. 我的应用程序使用 JRobin 1.4 ，这是我三年前在我的应用程序中编写的内容。
    
    
    2. 该应用程序最繁忙的实例目前创建
       
       
    1. My application records and displays data with JRobin 1.4, something I coded into my app over three years ago.
    2. The busiest instance of the application currently creates
    
    
    1. 启动后一小时内，1000多个新的JRobin数据库文件（每个大约1.3MB）
    
    
    2. ~100 +每天启动后
    li>
    
    1. Over 1000 a few new JRobin database files (at about 1.3MB each) within an hour of starting up
    2. ~100+ each day after start-up

  
  
  • 如果有什么要写的话，该应用每15秒更新一次这些JRobin数据库对象。
  
  
  • 在默认配置中JRobin：
    
    

  • The app updates these JRobin data base objects once every 15s, if there is something to write.
  • In the default configuration JRobin:
    
    
    1. 使用基于 java.nio 的文件访问后端。这个后端将 MappedByteBuffers 映射到文件本身。
    
    
    2. 每5分钟一次JRobin守护程序线程调用每个JRobin底层数据库MBB上的MappedByteBuffer.force（）
    
    
    1. uses a java.nio-based file access back-end. This back-end maps MappedByteBuffers to the files themselves.
    2. once every five minutes a JRobin daemon thread calls MappedByteBuffer.force() on every JRobin underlying database MBB

  
  
  • pmap 已列出：
    
    

  • pmap listed:
    
    
    1. 6500映射
    
    
    2. 5500其中包含1.3MB JRobin数据库文件， out to~7.1GB
    
    

  
  

  最后一点是我的Eureka ！时刻。

  我的纠正措施：

  My corrective actions:

  
  
  1. 考虑更新到最新的JRobinLite 1.5.2，这显然更好
  
  
  2. 在JRobin数据库上实施适当的资源处理。目前，一旦我的应用程序创建了一个数据库，然后在数据库不再被主动使用后再也不会转储它。
  
  
  3. 尝试移动 MappedByteBuffer.force（） 到数据库更新事件，而不是定期计时器。这个问题会不会神奇地消失？
  
  
  4. 立即，将JRobin后端更改为java.io实现 - 换行符。这会慢一点，但可能不是问题。以下是显示此更改的直接影响的图表。
  
  
  1. Consider updating to the latest JRobinLite 1.5.2 which is apparently better
  2. Implement proper resource handling on JRobin databases. At the moment, once my application creates a database and then never dumps it after the database is no longer actively used.
  3. Experiment with moving the MappedByteBuffer.force() to database update events, and not a periodic timer. Will the problem magically go away?
  4. Immediately, change the JRobin back-end to the java.io implementation--a line line change. This will be slower, but it is possibly not an issue. Here is a graph showing the immediate impact of this change.

  使用Java RSS内存图http://lanai.dietpizza.ch/images/stackoverflow-rss-problem-fixed.png

  我可能有或没有时间弄清楚的问题：

  Questions that I may or may not have time to figure out:

  
  
  • 使用 MappedByteBuffer.force（）在JVM内部发生了什么？如果没有任何改变，它是否仍然写入整个文件？部分文件？是否先加载它？
  
  
  • 始终在RSS中是否有一定数量的MBB？ （RSS大约是分配的MBB总数的一半。巧合？我怀疑没有。）
  
  
  • 如果我移动 MappedByteBuffer.force（）对于数据库更新事件而不是定期计时器，问题会不会消失？
  
  
  • 为什么RSS斜率如此规律？它与任何应用程序负载指标都没有关联。
  
  
  • What is going on inside the JVM with MappedByteBuffer.force()? If nothing has changed, does it still write the entire file? Part of the file? Does it load it first?
  • Is there a certain amount of the MBB always in RSS at all times? (RSS was roughly half the total allocated MBB sizes. Coincidence? I suspect not.)
  • If I move the MappedByteBuffer.force() to database update events, and not a periodic timer, will the problem magically go away?
  • Why was the RSS slope so regular? It does not correlate to any of the application load metrics.

  推荐答案

  只是一个想法：NIO缓冲区放在JVM之外。

  Just an idea: NIO buffers are placed outside the JVM.

  编辑：
  2016年值得考虑@Lari Hotari评论[为什么即使堆等大小稳定，Sun JVM也会继续消耗更多的RSS内存？因为回到2009年，RHEL4已经glibc< 2.10（~2.3）

  As per 2016 it's worth considering @Lari Hotari comment [ Why does the Sun JVM continue to consume ever more RSS memory even when the heap, etc sizes are stable? ] because back to 2009, RHEL4 had glibc < 2.10 (~2.3)

  问候。

  这篇关于为什么即使堆等大小稳定，Sun JVM也会继续消耗更多的RSS内存？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！