正常运行时间后，JDK7应用程序变慢 [英] JDK7 Application is getting slow after some Uptime

问题描述

我们在JBoss上部署了一个大型的JDK7应用程序，使用了一些库，例如Hibernate，Spring等.最初启动服务器后，该应用程序将按预期运行，但是在正常运行时间后，它将变得非常缓慢.

使用探查器，我们已经看到，每次out应用程序的某些方面变慢时，但并不总是相同的方面.在一次运行中，休眠刷新可能会缓慢进行爬网，而在另一次运行中，则可能是来自Spring的一些DI代码.

那里发生了什么事?

解决方案

JDK7中有一个关于CodeCache内存区域的错误，这非常非常困扰我们.

说明

Java基本上会启动，并使用即时编译(JIT)来在运行时仅编译字节码的必需部分.这使JVM可以在执行过程中反编译某些代码片段并重新编译它们.如果JVM确定，则会发生某个代码片段的初始编译不理想的情况. Oracle在JDK 7中引入了名为分层编译的功能，该功能使VM可以做到这一点.

JVM中的

编译后的代码存储在CodeCache内存区域中.直到JDK6为止，默认设置是该区域将被填充，并且一旦达到100％，JIT将停止编译并且将向控制台打印错误，但是该应用程序将与以前一样运行:已编译的所有内容将保持编译状态，所有尚未编译的内容都将以解释模式执行(速度大约慢100倍)

此选项名为CodeCacheFlushing，自JDK7u4起默认启用.这样的想法是，一旦CodeCache已满，就会从内存中清除最不常用的编译代码部分，以便为其他代码片段腾出空间.这将使JDK6-default-behaviour(停止所有编译)成为过时.它还允许使用较小的CodeCache区域(在JDK7中，默认情况下，CodeCache为48M，如果启用了分层编译，则为96M).

错误来了.在JDK7中，一旦CodeCache满了，JIT就停止了.接下来是CodeCache区域的刷新.而已.刷新完成后应重新启用JIT，但这不会发生.另外，控制台上没有警告打印.更糟:在禁用JIT之前，大约一半的已编译代码被丢弃.

与JDK6相反，在JDK6中，所有快速的东西都将保持快速状态，并且仅会解释新代码，而在JDK7中，您实际上会丢失已经编译和优化的代码！应用程序中所有突然出现问题的部分都将停止运行.剩下的机会是，应用程序的哪些部分速度变慢，这几乎不可能通过探查器跟踪该bugger:有时，用于刷新的休眠代码会减慢其速度，而其他时候，它的spring DI代码或您自己的应用程序代码会变慢. >

您受到影响了吗?

您可以使用探查器(JProfiler/YourKit)或JConsole(不会使用JVisualVM)来监视CodeCache内存区域的内存消耗.通常，代码缓存量committed将保持非常接近used量(例如，committed为23mb，已使用为22mb).在您的应用程序运行时，committed和used会上升，直到committed到达max.此时，used将急剧下降至max的1/2-2/3.之后，used将不再生长.那就是错误会打击您的地方.在JConsole中，它将如下所示:

为什么不是我而是所有其他人?

有可能，您正在使用JBoss.甲骨文很快发现有些事情不是应该的，并且默认情况下禁用了tiered compilation －但是Red Hat以其无限的智慧决定了，它更加了解并重新启用了它.基本上，我们的web应用程序可以在Weblogic上正常运行，并且只影响JBoss，因为没有分层编译(在weblogic中未启用)，CodeCache的增长是如此之小，即使经过数周的运行，我们也从未达到48mb的门槛.

我该怎么办?

首先，确定此bug是否击中了您.其次，使错误更难以损坏您.如果禁用CodeCacheFlushing，至少击中该错误不会使事情变得比以前更糟.停止tiered compilation可以减少您遇到该错误的可能性，就像增加可用的CodeCache-Memory的数量一样.

您始终可以尝试切换到JDK8，这似乎不受影响，并且如果CodeCache运行完整，您还可以在软件中实施监视以警告您.

TL; DR

• 在JDK 7中，永远不要启用分层编译(默认情况下已禁用，已在JBoss中启用)
在JBoss 7中
• 始终在standalone.conf中设置PRESERVE_JAVA_OPTS=true
• 始终禁用CodeCacheFlushing(-XX:-UseCodeCacheFlushing)
• 始终将足够数量的内存打包到CodeCache(-XX:ReservedCodeCacheSize=xxM)中.

We have a large JDK7 application deployed on JBoss using several libraries like Hibernate, Spring and so on. After initial startup of the server, the application runs as expected but after some uptime it becomes very slow.

Using a profiler, we have seen that every time certain aspects of out application are slowing down but not always the same aspects. While in one run it might be that hibernate flush slows to a crawl, in another run it might be some DI-code from Spring.

What's going on there?

解决方案

There is a bug in JDK7 regarding the CodeCache memory area which hit us very, very hard.

Explanation

Basically Java starts up and uses just in time compilation (JIT) to compile just the required parts of the bytecode during runtime. This enables the JVM to de- and recompile certain code fragments during execution. This happend, if the JVM determins, that an initial compilation of a certain code fragment is suboptimal. Oracle introduced a feature named tiered compilation in JDK 7 which allows the VM to do just that.

Compiled code in the JVM is stored in the CodeCache memory area. Up to JDK6 the default was that this area would be filled up and once at a 100% the JIT would stop compiling and an error would be printed to the console, however the application would be running same as before: Everything already compiled would stay compiled, everything not yet compiled would be executed in interpretation mode (which is roughly 100x slower)

This option is named CodeCacheFlushing, it is enabled by default since JDK7u4. The idea is, that once CodeCache is full, the least used parts of compiled code are flushed from memory to make room for other code fragments. That would make the JDK6-default-behaviour (to stop compilation all in all) obsolete. It also allowed for a much smaller CodeCache area (in JDK7 CodeCache is 48M by default/96M if tiered compilation is enabled).

Here comes the bug. In JDK7 once the CodeCache gets full, the JIT is stopped. Next comes the flushing of the CodeCache area. That's it. JIT should be reenabled after flushing is completed but that doesn't happen. Also, there is no warning printed to the console. Worse: prior to disabling the JIT roughly half of the already compiled code is thrown out.

In contrast to JDK6 where everything that was fast will stay fast and only new code will be interpreted, in JDK7 you actually lose already compiled and optimized code! All of the sudden parts of your application that performed well will stop doing so. It is left to chance, which parts of the application slow down, which makes tracking that bugger by profiler nearly impossible: At times the hibernate code for flushing slows down, at other times, its the spring DI code or your own appcode.

Are you affected?

You can use a profiler (JProfiler/YourKit) or JConsole (JVisualVM won't do) to monitor the memory-consumption of CodeCache memory area. Typically the CodeCache amount committed will stay very close to the used amount (say, committed is 23mb, used is 22mb). While your application runs, committed and used go up until committed reaches max. At that point used will drop sharply to 1/2 - 2/3 of max. After that, used whill no longer grow. That's where the bug will hit you. In JConsole, it will look like this:

Why me and not all the others?

Chances are, you are using JBoss. Oracle quickly found out that there are things not like they should be and disabled tiered compilation by default - yet Red Hat in its infinite wisdom decided, it knew better and reenabled it. Basically our webapp runs fine on Weblogic and only JBoss is affected, because without the tiered compilation (not enabled in weblogic) the growth of CodeCache is so small, we never actually hit the 48mb threshold even after weeks of operation.

What can I do?

First of all, decide, whether this bug hits you. Second, make it harder for the bug to damage you. If you disable CodeCacheFlushing at least hitting the bug won't make things worse than they were before. Stopping tiered compilation will make it less probable the bug hits you, same as increasing the amount of CodeCache-Memory available.

You can always try to switch to JDK8, this seems unaffected and also you could implement monitoring in your software to warn you, if CodeCache is running full.

TL;DR

• In JDK 7 never enable tiered compilation (disabled by default, enabled in JBoss)
• in JBoss 7 always set PRESERVE_JAVA_OPTS=true in standalone.conf
• always disable CodeCacheFlushing (-XX:-UseCodeCacheFlushing)
• always pack a sufficient amount of memory into CodeCache (-XX:ReservedCodeCacheSize=xxM).

这篇关于正常运行时间后，JDK7应用程序变慢的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！