排除无限制的Java Resident Set Size（RSS）增长 [英] Troubleshooting unbounded Java Resident Set Size(RSS) growth

问题描述

我有一个独立的Java应用程序，它具有：

I have a standalone Java application which has:

-Xmx1024m -Xms1024m -XX:MaxPermSize=256m -XX:PermSize=256m

随着时间的推移，它会占用越来越多的内存，开始交换（并减速）并最终死了很多次（不是OOM +转储，只是死了，/ var / log / messages没什么）。

Over the course of time it hogs more and more memory, starts to swap(and slow down) and eventually died a number of times(not OOM+dump, just died, nothing on /var/log/messages).

到目前为止我尝试过的事情：

What I've tried so far:

1. 堆转储：活动对象从1G堆中取出200-300Mb - > ok with heap


2. 活动线程数相当不变（~60-70） - >确定线程堆栈


3. JMX在某个时刻停止回答（mb它回答但是超时更低）


4. 关闭交换 - 它死得更快


5. strace - 似乎一切都变慢了，应用仍然没有死，并且不确定哪些东西看起来那里


6. 检查顶部：VIRT增长到5.5Gb，RSS增加到3.7 Gb



7. 检查vmstat（显然我们开始交换）：

1. Heap dumps: live objects take 200-300Mb out of 1G heap --> ok with heap
2. Number of live threads is rather constant(~60-70) --> ok with thread stacks
3. JMX stops answering at some point(mb it answers but timeout is lower)
4. Turn off swap - it dies faster
5. strace - seems everything slows down a bit, app still haven't died, and not sure for which things look there
6. Checking top: VIRT grows to 5.5Gb, RSS to 3.7 Gb

7. Checking vmstat(obviously we start to swap):

 --------------------------procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
Sun Jul 22 16:10:26 2012:  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
Sun Jul 22 16:48:41 2012:  0  0 138652 2502504  40360 706592    1    0   169    21 1047  206 20  1 74  4  0
. . . 
Sun Jul 22 18:10:59 2012:  0  0 138648  24816  58600 1609212    0    0   124   669  913 24436 43 22 34  2  0
Sun Jul 22 19:10:22 2012: 33  1 138644  33304   4960 1107480    0    0   100   536  810 19536 44 22 23 10  0
Sun Jul 22 20:10:28 2012: 54  1 213916  26928   2864 578832    3  360   100   710  639 12702 43 16 30 11  0
Sun Jul 22 21:10:43 2012:  0  0 629256  26116   2992 467808   84  176   278  1320 1293 24243 50 19 29  3  0
Sun Jul 22 22:10:55 2012:  4  0 772168  29136   1240 165900  203   94   435  1188 1278 21851 48 16 33  2  0
Sun Jul 22 23:10:57 2012:  0  1 2429536  26280   1880 169816 6875 6471  7081  6878 2146 8447 18 37  1 45  0

sar还显示稳定的系统％增长=交换：

sar also shows steady system% growth = swapping:

 15:40:02          CPU     %user     %nice   %system   %iowait    %steal     %idle
 17:40:01          all     51.00      0.00      7.81      3.04      0.00     38.15
 19:40:01          all     48.43      0.00     18.89      2.07      0.00     30.60
 20:40:01          all     43.93      0.00     15.84      5.54      0.00     34.70
 21:40:01          all     46.14      0.00     15.44      6.57      0.00     31.85
 22:40:01          all     44.25      0.00     20.94      5.43      0.00     29.39
 23:40:01          all     18.24      0.00     52.13     21.17      0.00      8.46
 12:40:02          all     22.03      0.00     41.70     15.46      0.00     20.81

检查pmap是否有以下最大贡献者：

Checking pmap gaves the following largest contributors:

  000000005416c000 1505760K rwx--    [ anon ]
  00000000b0000000 1310720K rwx--    [ anon ]
  00002aaab9001000 2079748K rwx--    [ anon ]

尝试关联我从strace转储的pmap中得到的地址给了我没有匹配

Trying to correlate addresses I've got from pmap from stuff dumped by strace gave me no matches

添加更多内存是不切实际的（稍后会出现问题）

Adding more memory is not practical(just make problem appear later)

问题是：
还有什么我可以尝试追查问题的原因或尝试解决它？

推荐答案

问题在于附加了一个探查器库 - 它记录了CPU调用/分配站点，因此需要内存来存储它。

Problem was in a profiler library attached - it recorded CPU calls/allocation sites, thus required memory to store that.

所以，人为因素这里：）

这篇关于排除无限制的Java Resident Set Size（RSS）增长的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！