多个Grok过滤器未存储第一个过滤器匹配记录 [英] Multiple Grok Filters not storing first filter match record

问题描述

我正在使用Logstash解析后缀日志.我主要致力于从后缀日志中获取退回的电子邮件日志，并将其存储在数据库中.

I am using Logstash to parse postfix logs. I am mainly focused to get bounced email logs from postfix logs, and store it in database.

为了获取日志，首先我需要找到由postfix生成的与我的message-id相对应的ID，然后使用该ID，我需要查找电子邮件的状态.对于以下配置，我能够获取日志.

In order to get logs, first I need to find ID generated by postfix corresponding to my message-id, and using that Id, I need to find status of an email. For following configuation, I am able to get the logs.

grok {
       patterns_dir => "patterns"
       match => [
            "message", "%{SYSLOGBASE} %{POSTFIXCLEANUP}",
            "message", "%{SYSLOGBASE} %{POSTFIXBOUNCE}"
        ]
        named_captures_only => true
    }

我使用以下if条件存储与模式匹配的日志:

I am using following if condition to store logs that match the patterns:

if "_grokparsefailure" not in [tags] {
   #database call
}

如您所见，我正在使用两种模式从一个日志文件中找到相应的两个不同的日志.

As you have seen, I am using two patterns to find corresponding two different logs from one log file.

现在，我想根据标签来区分这两种模式.因此，我对配置进行了如下修改:

Now, I want to differentiate both pattern based on tags. So I have modified my configuration as follows:

  grok {
       patterns_dir => "patterns"
       match => [
            "message", "%{SYSLOGBASE} %{POSTFIXBOUNCE}"
        ]
        add_tag => ["BOUNCED"]
        remove_tag => ["_grokparsefailure"]
        named_captures_only => true
    }

    grok {
       patterns_dir => "patterns"
       match => [
            "message", "%{SYSLOGBASE} %{POSTFIXCLEANUP}"            
        ]
        add_tag => ["INTIALIZATION"]
        remove_tag => ["_grokparsefailure"]
        named_captures_only => true
    }

现在，它仅存储％{POSTFIXCLEANUP}个模式日志.如果我颠倒顺序，它只会存储％{POSTFIXBOUNCE}模式.

Now, it only store %{POSTFIXCLEANUP} pattern logs. If I reverse the order, it only store %{POSTFIXBOUNCE} pattern.

因此，删除条件条件后，我发现从第一个过滤器解析的消息具有"_grokparsefailure"标签和第一个过滤器标签，因此该消息未存储该记录.

so, after removing that if condition, I found that message being parsed from first filter have "_grokparsefailure" tag and first filter tag, and because of that it is not storing that record.

有人可以告诉我需要做些什么来纠正此问题吗?我有什么错误吗?

Can anybody tell me what need to be done to rectify this? Am I am making any mistake?

推荐答案

您需要保护第二个grok块-即，如果第一个成功，则不要执行它.

You need to protect the 2nd grok block -- ie don't execute it if the first one succeeds.

if ("BOUNCED" not in [tags]) {
  grok {
    patterns_dir => "patterns"
    match => [
        "message", "%{SYSLOGBASE} %{POSTFIXCLEANUP}"            
    ]
    add_tag => ["INTIALIZATION"]
    remove_tag => ["_grokparsefailure"]
    named_captures_only => true
  }
}

这篇关于多个Grok过滤器未存储第一个过滤器匹配记录的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！