忽略第三方库的重复异常消息的日志记录 [英] Ignore logging of duplicated exception messages of third party libs

问题描述

我需要处理日志中特定例外的重复.

I need to handle duplication of specific exeptions in my logs.

我使用 slf4j 和 logback 来登录我的应用程序.我使用了一些外部服务(DB、apache kafka、第三方库等).当与此类服务的连接丢失时，我遇到了异常，例如

I use slf4j and logback for logging in my application. I use some external services(DB, apache kafka, third-party libs, etc.). When connection was lost to such service I got exception, such as

[kafka-producer-network-thread | producer-1] WARN  o.a.kafka.common.network.Selector - Error in I/O with localhost/127.0.0.1
java.net.ConnectException: Connection refused: no further information
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_45]
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[na:1.8.0_45]
    at org.apache.kafka.common.network.Selector.poll(Selector.java:238) ~[kafka-clients-0.8.2.0.jar:na]
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:192) [kafka-clients-0.8.2.0.jar:na]
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:191) [kafka-clients-0.8.2.0.jar:na]
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:122) [kafka-clients-0.8.2.0.jar:na]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]

问题是我每秒都会收到这条消息.此异常消息将淹没我的日志文件，因此 N 小时后我将在日志文件中保留几 GB.

The problem is that I got this message every second. This exception message will flood my log file, so I'll have a few GBs in log file in N hours.

我希望每 1-5 分钟有一次关于此异常的日志消息.有没有办法忽略日志文件中的重复异常?

可能的解决方案:

1. 忽略特定包和类的所有日志.[糟糕，因为我可以跳过重要消息]

1. Ignore all logs for specific package and class. [bad, beucase I can skip important message]

使用 http://logback.qos.ch/manual/filters.html#DuplicateMessageFilter[不好，因为我只能设置 AllowedRepetitions 或 CacheSize 属性.它将匹配所有消息，但我只需要特定的例外]

Use http://logback.qos.ch/manual/filters.html#DuplicateMessageFilter [bad, because I can set only properties AllowedRepetitions or CacheSize. It'll match to all messages, but I need only specific exeption]

编写自定义过滤器也许，您知道已经实现的解决方案?

Write custom filter Maybe, you know already imlemented solutions?

推荐答案

编写新的 turbo 过滤器并实现任何逻辑来拒绝某些特定日志记录真的很容易.

It's really easy to write new turbo filter and implement any logic to reject some specific logging.

我用下一个配置添加了新过滤器logback.xml

I've ilemented new filter with next configuration logback.xml

<turboFilter class="package.DuplicationTimeoutTurboFilter">
    <MinutesToBlock>3</MinutesToBlock>
    <KeyPattern>
        <loggerClass>org.apache.kafka.common.network.Selector</loggerClass>
        <message>java.net.ConnectException: Connection refused: no further information</message>
    </KeyPattern>
</turboFilter>

和实施:

import ch.qos.logback.classic.Level;
import ch.qos.logback.classic.Logger;
import ch.qos.logback.classic.turbo.TurboFilter;
import ch.qos.logback.core.spi.FilterReply;
import org.slf4j.Marker;

import java.time.LocalDateTime;
import java.util.Arrays;
import java.util.HashSet;
import java.util.Objects;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.stream.Collectors;

public class DuplicationTimeoutTurboFilter extends TurboFilter {

    private static final int CLEAN_UP_THRESHOLD = 1000;
    private ConcurrentHashMap<KeyPattern, LocalDateTime> recentlyMatchedPatterns = new ConcurrentHashMap<>();

    private Set<KeyPattern> ignoringPatterns = new HashSet<>();

    private long minutesToBlock = 3L;

    @Override
    public FilterReply decide(Marker marker, Logger logger, Level level, String format, Object[] params, Throwable t) {
        String rawLogMessage = format + Arrays.toString(params) + Objects.toString(t);  //sometimes throwable can be inserted into params argument

        Set<KeyPattern> matchedIgnoringSet = ignoringPatterns.stream()
                .filter(key -> match(key, logger, rawLogMessage))
                .collect(Collectors.toSet());

        if (!matchedIgnoringSet.isEmpty() && isLoggedRecently(matchedIgnoringSet)) {
            return FilterReply.DENY;
        }

        return FilterReply.NEUTRAL;
    }

    private boolean match(KeyPattern keyPattern, Logger logger, String rawText) {
        String loggerClass = keyPattern.getLoggerClass();
        String messagePattern = keyPattern.getMessage();
        return loggerClass.equals(logger.getName()) && rawText.contains(messagePattern);
    }

    private boolean isLoggedRecently(Set<KeyPattern> matchedIgnoredList) {
        for (KeyPattern pattern : matchedIgnoredList) {
            LocalDateTime now = LocalDateTime.now();

            LocalDateTime lastLogTime = recentlyMatchedPatterns.putIfAbsent(pattern, now);
            if (lastLogTime == null) {
                return false;
            }

            LocalDateTime blockedTillTime = lastLogTime.plusMinutes(minutesToBlock);
            if (blockedTillTime.isAfter(now)) {
                return true;
            } else if (blockedTillTime.isBefore(now)) {
                recentlyMatchedPatterns.put(pattern, now);
                cleanupIfNeeded();
                return false;
            }
        }
        return true;
    }

    private void cleanupIfNeeded() {
        if (recentlyMatchedPatterns.size() > CLEAN_UP_THRESHOLD) {
            LocalDateTime oldTime = LocalDateTime.now().minusMinutes(minutesToBlock * 2);
            recentlyMatchedPatterns.values().removeIf(lastLogTime -> lastLogTime.isAfter(oldTime));
        }
    }

    public long getMinutesToBlock() {
        return minutesToBlock;
    }

    public void setMinutesToBlock(long minutesToBlock) {
        this.minutesToBlock = minutesToBlock;
    }

    public void addKeyPattern(KeyPattern keyPattern) {
        ignoringPatterns.add(keyPattern);
    }

    public static class KeyPattern {
        private String loggerClass;
        private String message;

        //constructor, getters, setters, equals, hashcode
    }
}

这篇关于忽略第三方库的重复异常消息的日志记录的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！