如何处理不匹配的 Logstash grok 过滤器 [英] How to handle non-matching Logstash grok filters

问题描述

我想知道使用我的 Logstash Grok 过滤器的最佳方法是什么.我有一些用于特定日志条目的过滤器，不会应用于所有条目.不适用的总是生成 _grokparsefailure 标签.例如，我有一个适用于每个日志条目的 grok 过滤器，它运行良好.然后我有另一个过滤器，用于带有回溯的错误消息.回溯过滤器会为每个没有回溯的日志条目抛出 grokparsefailure.

I am wondering what the best approach to take with my Logstash Grok filters. I have some filters that are for specific log entries, and won't apply to all entries. The ones that don't apply always generate _grokparsefailure tags. For example, I have one grok filter that's for every log entry and it works fine. Then I have another filter that's for error messages with tracebacks. The traceback filter throws a grokparsefailure for every single log entry that doesn't have a traceback.

如果没有匹配项，我宁愿让它通过规则，而不是添加 parsefailure 标记.我使用 parsefailure 标记来查找未正确解析的内容，而不是根本不匹配特定过滤器的内容.也许这只是让我着迷的术语解析失败".对我来说，这意味着过滤器有问题(例如格式错误)，而不是不匹配.

I'd prefer to have it just pass the rule if there isn't a match instead of adding the parsefailure tag. I use the parsefailure tag to find things that aren't parsing properly, not things that simply didn't match a particular filter. Maybe it's just the nomenclature "parse failure" that gets me. To me that means there's something wrong with the filter (e.g. badly formatted), not that it didn't match.

那么问题来了，我该如何处理?

So the question is, how should I handle this?

• 使用 ?

• Make the filter pattern optional using ?

(ab) 通过将 tag_on_failure 选项设置为空来使用 []

(ab)use the tag_on_failure option by setting it to nothing []

使用诸如if traceback in message"之类的东西使过滤器有条件

make the filter conditional using something like "if traceback in message"

还有什么我不考虑的吗?

something else I'm not considering?

提前致谢.

编辑

我采取了在过滤器周围添加条件的路径:

I took the path of adding a conditional around the filter:

    if [message] =~ /tooksd+/ {
        grok {
            patterns_dir => "/etc/logstash/patterns"
            match => ["message", "tooks+(?<servicetime>[d.]+)"]
            add_tag => [ "stats", "servicetime" ]
        }
    }

不过仍然对反馈感兴趣.什么是此处的最佳实践"?

Still interested in feedback though. What is considered "best practice" here?

推荐答案

如果可能，我会选择 条件包装器 就像你正在使用的那个.随意将其发布为答案！

When possible, I'd go with a conditional wrapper just like the one you're using. Feel free to post that as an answer!

如果您的应用程序只生成几种不同的行格式，您可以使用多个匹配模式和 grok 过滤器.默认情况下，过滤器将处理到第一个成功匹配:

If your application produces only a few different line formats, you can use multiple match patterns with the grok filter. By default, the filter will process up to the first successful match:

grok {
    patterns_dir => "./patterns"
    match => {
        "message" => [ 
              "%{BASE_PATTERN} %{EXTRA_PATTERN}",
              "%{BASE_PATTERN}",
              "%{SOME_OTHER_PATTERN}"
        ]
    }
}

如果您的逻辑不那么简单(可能您需要多次检查相同的条件)，grep 过滤器 可用于添加标签.像这样:

If your logic is less straightforward (maybe you need to check the same condition more than once), the grep filter can be useful to add a tag. Something like this:

grep {
    drop => false #grep normally drops non-matching events
    match => ["message", "/tooksd+/"]
    add_tag => "has_traceback"
}


...

if "has_traceback" in [tags] {
    ...
}

这篇关于如何处理不匹配的 Logstash grok 过滤器的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！