寿命测试中高负载下的JVM Freeze [英] JVM Freeze under high load in longevity tests

问题描述

使用JVM运行:

java version "1.7.0_79"
Java(TM) SE Runtime Environment (build 1.7.0_79-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)

操作系统:

CentOS release 6.4 (Final)

Jvm选项:

-Xmx4g -Xms4g -XX:MaxPermSize=4g -XX:+HeapDumpOnOutOfMemoryError -XX:+PrintClassHistogram -XX:+CMSClassUnloadingEnabled -verbose:gc -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+DisableExplicitGC

在OSGI环境中运行，用于网络的Aerospike DB，NETTY(NIO).

Running in an OSGI environment, Aerospike DB, NETTY (NIO) for networking.

进行周末寿命测试. 这是最后的印刷品:

Ran a weekend longevity test. This was the last print:

[2015-12-11 09:54:51,185] INFO  : [GC pause (young)

两天后，我在pid上运行了strace，然后接下来是这些打印:

After 2 days I ran strace on the pid, and then those are the next prints:

[2015-12-11 09:54:51,185] INFO  : [GC pause (young) 3598M->1458M(4096M), 0.0280020 secs]
[2015-12-13 11:54:54,353] INFO  : [GC pause (young) 3598M->1464M(4096M), 180001.5628870 secs]

第一张打印完成，下一张显示GC为期2天.

The first print finished and the next print showed a 2 days GC.

在冻结期间，jvm没有响应线程转储信号(pkill -QUIT pid). 这种冻结每隔几天发生一次. 冻结不仅发生在G1收集器上，还发生在CMS收集器上. 我该如何开始调试它，以及可能导致此问题的原因?

The jvm did not respone to thread dump signals during the freeze (pkill -QUIT pid). This freeze happens every few days. The freeze happens not only with the G1 collector, but also with CMS collector. How can I start debugging this, and what can potentially cause this?

谢谢.

再次冻结，这一次strace不会释放冻结. 第二次冻结是使用jstack释放的.

Had another freeze, this time the strace does not release the freeze. The second freeze was released using jstack.

更新: 发现了问题！请看下面的答案.

UPDATE: Found the problem! Look at the answer below.

推荐答案

我发现了问题！
这是futex_wait()中的一个内核错误，已被反向移植到我们的内核版本.
您可以在这里阅读有关信息:
https://groups.google.com/forum/#!topic/mechanical -sympathy/QbmpZxp6C64

I found the problem!
It is a kernel bug in futex_wait() that was backported to our kernel version.
You can read about it here:
https://groups.google.com/forum/#!topic/mechanical-sympathy/QbmpZxp6C64

这篇关于寿命测试中高负载下的JVM Freeze的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！