忽略记录第三方库的重复异常消息 [英] Ignore logging of duplicated exception messages of third party libs

问题描述

我需要处理日志中特定片段的重复.

I need to handle duplication of specific exeptions in my logs.

我使用slf4j和logback登录我的应用程序.我使用一些外部服务(数据库，apache kafka，第三方库等).当失去与此类服务的连接时，我遇到了异常情况，例如

I use slf4j and logback for logging in my application. I use some external services(DB, apache kafka, third-party libs, etc.). When connection was lost to such service I got exception, such as

[kafka-producer-network-thread | producer-1] WARN  o.a.kafka.common.network.Selector - Error in I/O with localhost/127.0.0.1
java.net.ConnectException: Connection refused: no further information
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[na:1.8.0_45]
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[na:1.8.0_45]
    at org.apache.kafka.common.network.Selector.poll(Selector.java:238) ~[kafka-clients-0.8.2.0.jar:na]
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:192) [kafka-clients-0.8.2.0.jar:na]
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:191) [kafka-clients-0.8.2.0.jar:na]
    at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:122) [kafka-clients-0.8.2.0.jar:na]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]

问题是我每秒收到此消息.此异常消息将淹没我的日志文件，因此我将在N小时内在日志文件中保留几GB.

The problem is that I got this message every second. This exception message will flood my log file, so I'll have a few GBs in log file in N hours.

我希望每1-5分钟收到一次有关此异常的日志消息.有什么方法可以忽略日志文件中的重复异常吗?

可能的解决方案:

1. 忽略特定程序包和类的所有日志.[不好，因为我可以跳过重要的消息]

1. Ignore all logs for specific package and class. [bad, beucase I can skip important message]

使用 http://logback.qos.ch/manual/filters.html#DuplicateMessageFilter [不好，因为我只能设置属性AllowedRepetitions或CacheSize.它会匹配所有消息，但我只需要特定的字母]

Use http://logback.qos.ch/manual/filters.html#DuplicateMessageFilter [bad, because I can set only properties AllowedRepetitions or CacheSize. It'll match to all messages, but I need only specific exeption]

编写自定义过滤器也许，您知道已经实施的解决方案了吗?

Write custom filter Maybe, you know already imlemented solutions?

推荐答案

编写新的Turbo过滤器并实施任何逻辑来拒绝某些特定的日志记录确实很容易.

It's really easy to write new turbo filter and implement any logic to reject some specific logging.

我在下一个配置中使用了新的过滤器 logback.xml

I've ilemented new filter with next configuration logback.xml

<turboFilter class="package.DuplicationTimeoutTurboFilter">
    <MinutesToBlock>3</MinutesToBlock>
    <KeyPattern>
        <loggerClass>org.apache.kafka.common.network.Selector</loggerClass>
        <message>java.net.ConnectException: Connection refused: no further information</message>
    </KeyPattern>
</turboFilter>

和实现:

import ch.qos.logback.classic.Level;
import ch.qos.logback.classic.Logger;
import ch.qos.logback.classic.turbo.TurboFilter;
import ch.qos.logback.core.spi.FilterReply;
import org.slf4j.Marker;

import java.time.LocalDateTime;
import java.util.Arrays;
import java.util.HashSet;
import java.util.Objects;
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import java.util.stream.Collectors;

public class DuplicationTimeoutTurboFilter extends TurboFilter {

    private static final int CLEAN_UP_THRESHOLD = 1000;
    private ConcurrentHashMap<KeyPattern, LocalDateTime> recentlyMatchedPatterns = new ConcurrentHashMap<>();

    private Set<KeyPattern> ignoringPatterns = new HashSet<>();

    private long minutesToBlock = 3L;

    @Override
    public FilterReply decide(Marker marker, Logger logger, Level level, String format, Object[] params, Throwable t) {
        String rawLogMessage = format + Arrays.toString(params) + Objects.toString(t);  //sometimes throwable can be inserted into params argument

        Set<KeyPattern> matchedIgnoringSet = ignoringPatterns.stream()
                .filter(key -> match(key, logger, rawLogMessage))
                .collect(Collectors.toSet());

        if (!matchedIgnoringSet.isEmpty() && isLoggedRecently(matchedIgnoringSet)) {
            return FilterReply.DENY;
        }

        return FilterReply.NEUTRAL;
    }

    private boolean match(KeyPattern keyPattern, Logger logger, String rawText) {
        String loggerClass = keyPattern.getLoggerClass();
        String messagePattern = keyPattern.getMessage();
        return loggerClass.equals(logger.getName()) && rawText.contains(messagePattern);
    }

    private boolean isLoggedRecently(Set<KeyPattern> matchedIgnoredList) {
        for (KeyPattern pattern : matchedIgnoredList) {
            LocalDateTime now = LocalDateTime.now();

            LocalDateTime lastLogTime = recentlyMatchedPatterns.putIfAbsent(pattern, now);
            if (lastLogTime == null) {
                return false;
            }

            LocalDateTime blockedTillTime = lastLogTime.plusMinutes(minutesToBlock);
            if (blockedTillTime.isAfter(now)) {
                return true;
            } else if (blockedTillTime.isBefore(now)) {
                recentlyMatchedPatterns.put(pattern, now);
                cleanupIfNeeded();
                return false;
            }
        }
        return true;
    }

    private void cleanupIfNeeded() {
        if (recentlyMatchedPatterns.size() > CLEAN_UP_THRESHOLD) {
            LocalDateTime oldTime = LocalDateTime.now().minusMinutes(minutesToBlock * 2);
            recentlyMatchedPatterns.values().removeIf(lastLogTime -> lastLogTime.isAfter(oldTime));
        }
    }

    public long getMinutesToBlock() {
        return minutesToBlock;
    }

    public void setMinutesToBlock(long minutesToBlock) {
        this.minutesToBlock = minutesToBlock;
    }

    public void addKeyPattern(KeyPattern keyPattern) {
        ignoringPatterns.add(keyPattern);
    }

    public static class KeyPattern {
        private String loggerClass;
        private String message;

        //constructor, getters, setters, equals, hashcode
    }
}

这篇关于忽略记录第三方库的重复异常消息的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！