一个用于确定JVM何时将进入内存/ GC故障的有用指标 [英] A useful metric for determining when the JVM is about to get into memory/GC trouble

问题描述

我有一个scala数据处理应用程序，95％的时间可以处理内存中抛出的数据。剩下的5％如果不选中，通常不会触发OutOfMemoryError，但是会进入一个主循环周期的主要GC，它们会阻止后台线程执行，并且如果完成了，则需要10倍-50x，只要它有足够的内存。

我已经实现了可以将数据刷新到磁盘并将磁盘流视为内存迭代器。它通常比内存慢一个数量级，但对于这5％的情况就足够了。目前，我正在通过追踪数据处理中涉及的各种集合的大小的集合上下文的最大大小的启发式触发。这很有效，但实际上这只是一个特殊的经验门槛。

我当时对JVM接近上述不良状态并刷新到磁盘的情况反应良好。我尝试过观看记忆，但无法找到伊甸园，老年人等的可靠预测死亡螺旋的正确组合。我也尝试过观察主要地理信息系统的频率，但这似乎也受到太保守和太晚的广泛影响。

任何对于判断JVM运行状况和检测故障状态的资源将不胜感激。

解决方案

一种可靠的方法是在GC事件上注册通知侦听器，并在所有Full GC事件后检查内存运行状况。直接在完整的GC事件之后，使用的内存就是您实际的实时数据集。如果您在那个时间点的空闲内存不足，它可能是时间开始流向磁盘。

这样可以避免在尝试检查内存，不知道何时发生完整的GC，例如使用 MEMORY_THRESHOLD_EXCEEDED 通知类型时。

您可以注册一个通知侦听器，并使用如下代码处理完整的GC事件：

  // ... ...标准导入省略
 import com.sun.management.GarbageCollectionNotificationInfo; 
 
 public static void installGCMonitoring（）{
 List< GarbageCollectorMXBean> gcBeans = ManagementFactory.getGarbageCollectorMXBeans（）; （GarbageCollectorMXBean gcBean：gcBeans）{
 NotificationEmitter emitter =（NotificationEmitter）gcBean; 
 NotificationListener listener = notificationListener（）; 
 emitter.addNotificationListener（listener，null，null）; 
 
 
 
 private static NotificationListener notificationListener（）{
 return new NotificationListener（）{
 @Override 
 public void handleNotification（Notification通知，Object handback）{
 if（notification.getType（）
 .equals（GarbageCollectionNotificationInfo.GARBAGE_COLLECTION_NOTIFICATION））{
 GarbageCollectionNotificationInfo info = GarbageCollectionNotificationInfo $ b $ .from（（CompositeData）通知。 getUserData（））; 
 String gctype = info.getGcAction（）; 
 if（gctype.contains（major））{
 //我们只对全部（主要）GCs感兴趣
 Map< String，MemoryUsage> mem = info.getGcInfo（）。getMemoryUsageAfterGc（）; 
 for（Entry< String，MemoryUsage> entry：mem.entrySet（））{
 String memoryPoolName = entry.getKey（）; 
 MemoryUsage memdetail = entry.getValue（）; 
 long memMax = memdetail.getMax（）; 
 long memUsed = memdetail.getUsed（）; 
 //使用您感兴趣的池（可能是旧版本）
 //的memMax / memUsed来确定内存的健康状况。 
} 
} 
} 
} 
}; 
} 
  

债权至这篇文章是我们首先从中获得的。

I have a scala data processing application that 95% of the time can handle the data thrown at it in memory. The remaining 5% if left unchecked doesn't usually hit OutOfMemoryError, but just gets into a cycle of major GCs that spikes the CPU, prevents background threads from executing and, if it does even finish, takes 10x-50x as long as when it has enough memory.

I've implemented system that can flush data to disk and treat the disk stream as if it was an in-memory iterator. It's usually an order of magnitude slower than memory, but sufficient for these 5% cases. I'm currently triggering by a heuristic of max size of a collection context that tracks the size of various collections involved in the data processing. This works, but really is just an adhoc empirical threshold.

I would much rather react to the JVM getting near the above bad state and flush to disk at that time. I've tried watching memory, but can't find the right combination of eden, old, etc. to reliably predict the death spiral. I've also tried just watching for frequency of major GCs but that also seems to suffer from having a wide range of "too conservative" to "too late".

Any resources for judging JVM health and detecting trouble states would be appreciated.

解决方案

One reliable way is to register a notification listener on GC events and check the memory health after all Full GC events. Directly after a full GC event, the memory used is your actual live set of data. If you at that point in time are low on free memory it is probably time start flusing to disk.

This way you can avoid false positives that often happens when you try to check memory with no knowledge of when a full GC has occurred, for example when using the MEMORY_THRESHOLD_EXCEEDED notification type.

You can register a notification listener and handle Full GC events using something like the following code:

// ... standard imports ommitted
import com.sun.management.GarbageCollectionNotificationInfo;

public static void installGCMonitoring() {
    List<GarbageCollectorMXBean> gcBeans = ManagementFactory.getGarbageCollectorMXBeans();
    for (GarbageCollectorMXBean gcBean : gcBeans) {
        NotificationEmitter emitter = (NotificationEmitter) gcBean;
        NotificationListener listener = notificationListener();
        emitter.addNotificationListener(listener, null, null);
    }
}

private static NotificationListener notificationListener() {
    return new NotificationListener() {
        @Override
        public void handleNotification(Notification notification, Object handback) {
            if (notification.getType()
                    .equals(GarbageCollectionNotificationInfo.GARBAGE_COLLECTION_NOTIFICATION)) {
                GarbageCollectionNotificationInfo info = GarbageCollectionNotificationInfo
                        .from((CompositeData) notification.getUserData());
                String gctype = info.getGcAction();
                if (gctype.contains("major")) {
                    // We are only interested in full (major) GCs
                    Map<String, MemoryUsage> mem = info.getGcInfo().getMemoryUsageAfterGc();
                    for (Entry<String, MemoryUsage> entry : mem.entrySet()) {
                        String memoryPoolName = entry.getKey();
                        MemoryUsage memdetail = entry.getValue();
                        long memMax = memdetail.getMax();
                        long memUsed = memdetail.getUsed();
                        // Use the memMax/memUsed of the pool you are interested in (probably old gen)
                        // to determine memory health.
                    }
                }
            }
        }
    };
}

Cred to this article where we first got this idea from.

这篇关于一个用于确定JVM何时将进入内存/ GC故障的有用指标的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！