用于确定 JVM 何时将陷入内存/GC 问题的有用指标 [英] A useful metric for determining when the JVM is about to get into memory/GC trouble

问题描述

我有一个 Scala 数据处理应用程序，它 95% 的时间可以处理内存中抛出的数据.剩下的 5% 如果不加以检查，通常不会遇到 OutOfMemoryError，而只是进入一个主要的 GC 循环，这会使 CPU 达到峰值，阻止后台线程执行，如果它完成了，则需要 10 倍-50x 只要它有足够的内存.

I have a scala data processing application that 95% of the time can handle the data thrown at it in memory. The remaining 5% if left unchecked doesn't usually hit OutOfMemoryError, but just gets into a cycle of major GCs that spikes the CPU, prevents background threads from executing and, if it does even finish, takes 10x-50x as long as when it has enough memory.

我已经实现了可以将数据刷新到磁盘并将磁盘流视为内存中迭代器的系统.它通常比内存慢一个数量级，但对于这 5% 的情况来说已经足够了.我目前正在通过收集上下文的最大大小的启发式触发，该启发式跟踪数据处理中涉及的各种集合的大小.这可行，但实际上只是一个临时的经验阈值.

I've implemented system that can flush data to disk and treat the disk stream as if it was an in-memory iterator. It's usually an order of magnitude slower than memory, but sufficient for these 5% cases. I'm currently triggering by a heuristic of max size of a collection context that tracks the size of various collections involved in the data processing. This works, but really is just an adhoc empirical threshold.

我宁愿对 JVM 接近上述不良状态做出反应，并在那时刷新到磁盘.我试过观察记忆，但找不到正确的伊甸园、旧的等组合来可靠地预测死亡螺旋.我也尝试过只关注主要 GC 的频率，但这似乎也受到范围广泛的过于保守"到太晚"的影响.

I would much rather react to the JVM getting near the above bad state and flush to disk at that time. I've tried watching memory, but can't find the right combination of eden, old, etc. to reliably predict the death spiral. I've also tried just watching for frequency of major GCs but that also seems to suffer from having a wide range of "too conservative" to "too late".

任何用于判断 JVM 健康状况和检测故障状态的资源都将不胜感激.

Any resources for judging JVM health and detecting trouble states would be appreciated.

推荐答案

一种可靠的方法是在 GC 事件上注册一个通知侦听器，并在所有 Full GC 事件后检查内存健康状况.在一个完整的 GC 事件之后，使用的内存是您实际的实时数据集.如果您当时的可用内存不足，则可能是时候开始刷新到磁盘了.

One reliable way is to register a notification listener on GC events and check the memory health after all Full GC events. Directly after a full GC event, the memory used is your actual live set of data. If you at that point in time are low on free memory it is probably time start flusing to disk.

这样您可以避免在不知道何时发生完全 GC 的情况下尝试检查内存时经常发生的误报，例如在使用 MEMORY_THRESHOLD_EXCEEDED 通知类型时.

This way you can avoid false positives that often happens when you try to check memory with no knowledge of when a full GC has occurred, for example when using the MEMORY_THRESHOLD_EXCEEDED notification type.

您可以使用以下代码注册通知侦听器并处理 Full GC 事件:

You can register a notification listener and handle Full GC events using something like the following code:

// ... standard imports ommitted
import com.sun.management.GarbageCollectionNotificationInfo;

public static void installGCMonitoring() {
    List<GarbageCollectorMXBean> gcBeans = ManagementFactory.getGarbageCollectorMXBeans();
    for (GarbageCollectorMXBean gcBean : gcBeans) {
        NotificationEmitter emitter = (NotificationEmitter) gcBean;
        NotificationListener listener = notificationListener();
        emitter.addNotificationListener(listener, null, null);
    }
}

private static NotificationListener notificationListener() {
    return new NotificationListener() {
        @Override
        public void handleNotification(Notification notification, Object handback) {
            if (notification.getType()
                    .equals(GarbageCollectionNotificationInfo.GARBAGE_COLLECTION_NOTIFICATION)) {
                GarbageCollectionNotificationInfo info = GarbageCollectionNotificationInfo
                        .from((CompositeData) notification.getUserData());
                String gctype = info.getGcAction();
                if (gctype.contains("major")) {
                    // We are only interested in full (major) GCs
                    Map<String, MemoryUsage> mem = info.getGcInfo().getMemoryUsageAfterGc();
                    for (Entry<String, MemoryUsage> entry : mem.entrySet()) {
                        String memoryPoolName = entry.getKey();
                        MemoryUsage memdetail = entry.getValue();
                        long memMax = memdetail.getMax();
                        long memUsed = memdetail.getUsed();
                        // Use the memMax/memUsed of the pool you are interested in (probably old gen)
                        // to determine memory health.
                    }
                }
            }
        }
    };
}

感谢这篇文章，我们最初是从那里得到这个想法的.

Cred to this article where we first got this idea from.

这篇关于用于确定 JVM 何时将陷入内存/GC 问题的有用指标的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！