Logstash grok 过滤器来标记接收和退回的消息 [英] Logstash grok filter to tag received and bounced messages

问题描述

Sthg 让我发疯了，我想解析 Postfix 日志以了解电子邮件的状态，这是我目前尝试的方法:

输入{文件{路径=>/var/log/mail.log"}}筛选 {千伏{修剪 =><>"}if [message] =~/[ "status=bounced" ]/{神通{patterns_dir =>/etc/logstash/patterns"匹配 =>{消息"=>"%{SYSLOGBASE} (?[0-9A-F]{10}): %{GREEDYDATA:message}"}add_tag =>弹跳"}}}输出 {如果[标签]中的弹跳"{标准输出 { 编解码器 =>红宝石调试 }}}

<小时>

ma​​il.log 示例:

7 月 26 日 04:18:34 mx12 postfix/cleanup[20659]:3mfHGL1r9gzyQP:message-id=<[e-mail]>>

7 月 26 日 04:18:34 mx12 postfix/smtp[20662]:3mfHGL1r9gzyQP:to=，relay=127.0.0.2[127.0.0.2]:25，delay=0.53，delays=0.1/0/0.23/0.16, dsn=2.0.0, status=sent/退回

<小时>

结果 1:

我向现有电子邮件地址发送电子邮件，mail.log 中的状态为:

<块引用>

已发送 (250 ok) : OKAY

但这是 Logstash 所说的:

.. 我看到每个 postfix 程序(qmgr、smtp、qmgr 再次..)生成的每条消息.换句话说，对于所有甚至不包含status=bounced"的消息.

然后我也试过:

 if [message] =~/[ "bounced" ]/{变异{add_tag =>[反弹"]}}if [message] =~/[ "message-id", "(.*)@www.mydomain.fr" ]/{变异 { add_tag =>[ 发送" ] }}神通{匹配 =>{消息"=>"%{SYSLOGBASE} (?[0-9A-F]{10}): %{GREEDYDATA:message}"}}

结果 2:Logstash 总是在这里添加 2 个标签:反弹 + 发送 :(

<小时>

预期结果:

我尝试做的是正是这个配置文件，但它是用旧版本的 Logstash 制作的(例如grep"现在不可用)，但是这正是我试图使工作:

http://tales.itnobody.com/2013/07/using-logstash-to-log-smtp-bounces-like-a-boss.html

一句话:

1. 任何带有 DSN 的条目 – 记录:QID、dsn
2. 任何匹配 message-id=< 的条目hashRegex > – 记录:QID，消息 ID

如下:

 输出{如果[标签]中的弹跳"{执行{命令 =>php -f/path/LogDSN.php %{QID} %{dsn} &"}}如果发送"在 [标签] {执行{命令 =>php -f/path/LogOutbound.php %{QID} %{message-id} &"}}}

但是我的过滤器有问题，这让我很抓狂，

有什么想法吗??

解决方案

我发现了问题.

它来自这个测试:

if [message] =~/[ "bounced" ]/{变异{add_tag =>[反弹"]}}

正则表达式是 / 之间的部分，所以你的正则表达式是这样评估的:https://regex101.com/r/eaB5jp/2

因此您的所有行都将匹配并获得标签.

为了工作，测试应该是:

if [message] =~/bounced/{变异{add_tag =>[反弹"]}}

Sthg makes me crazy, I would like to parse Postfix logs to know the status of emails, here is what I tried so far :

input {
   file {path => "/var/log/mail.log"}
}

filter {
    kv {
        trim => "<>"
    }

    if [message] =~ /[ "status=bounced" ]/ {
        grok {
            patterns_dir => "/etc/logstash/patterns"
            match => {"message" => "%{SYSLOGBASE} (?<QID>[0-9A-F]{10}): %{GREEDYDATA:message}"}
            add_tag => "bounce"
        }
    }

}
output {
   if "bounce" in [tags] {
      stdout { codec => rubydebug }
   }
}

---

Example of mail.log :

Jul 26 04:18:34 mx12 postfix/cleanup[20659]: 3mfHGL1r9gzyQP: message-id=<[e-mail]>

Jul 26 04:18:34 mx12 postfix/smtp[20662]: 3mfHGL1r9gzyQP: to=, relay=127.0.0.2[127.0.0.2]:25, delay=0.53, delays=0.13/0/0.23/0.16, dsn=2.0.0, status=sent / bounced

---

Result 1 :

I send an email to an existing email address, the status in mail.log is :

sent (250 ok) : OKAY

But here is what Logstash tells :

.. and I see that for every message generated by each postfix program (qmgr, smtp, qmgr again..). In other words, for all messages that even not contain "status=bounced".

Then I also tried :

   if [message] =~ /[ "bounced" ]/ {
     mutate {add_tag => [ "bounce" ]}
  }

  if [message] =~ /[ "message-id", "(.*)@www.mydomain.fr" ]/ {
      mutate { add_tag => [ "send" ] }
  }
  grok {
       match => {"message" => "%{SYSLOGBASE} (?<QID>[0-9A-F]{10}): %{GREEDYDATA:message}"}
  }

Result 2 : Logstash add here always 2 tags : bounce + send :(

---

Result expected :

What I try to do is exactly this config file, but it was made with an old version of Logstash ("grep" for example is not available now), but this is exactly what I try to make working :

http://tales.itnobody.com/2013/07/using-logstash-to-log-smtp-bounces-like-a-boss.html

In one word :

1. Any entries with a DSN – RECORD: QID, dsn
2. Any entries matching message-id=< hashRegex > – RECORD: QID, message-id

As follow :

    output{
       if "bounce" in [tags] {
           exec {
                command => "php -f /path/LogDSN.php %{QID} %{dsn} &"
           }
       }

       if "send" in [tags] {
           exec {
                command => "php -f /path/LogOutbound.php %{QID} %{message-id} &"
           }
       }
     }

But there is a problem in my filter, that makes me crazy,

Any idea ??

解决方案

I have found the problem.

It's coming from this test:

if [message] =~ /[ "bounced" ]/ {
  mutate {add_tag => [ "bounce" ]}
}

The regex is the part between the /, so your regex is evaluated like that : https://regex101.com/r/eaB5jp/2

So all your lines will match and get the tag.

In order to work, the test should be:

if [message] =~ /bounced/ {
  mutate {add_tag => [ "bounce" ]}
}

这篇关于Logstash grok 过滤器来标记接收和退回的消息的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！