Logstash：在事件之间保持一个值 [英] Logstash: Keeping a value across events

问题描述

我有一个日期只在每个日志文件中存在一次，并且我尝试在匹配一次之后将所有后续事件添加到所有后续事件中，使其在某些方面像全局变量一样。 （日期在文档的顶部，我无法使用 multiline 或更改文件名或内容）

为此，我的方法是使用 grep 过滤器与 drop =>假。

  grok {
 patterns_dir => [...]
 match => [message，％{DATELINE}] 
 tag_on_failure => [] 
} 
 grep {
 add_field => {grepdate=> ％{mydate}} 
 drop => false 
} 
 date {
 locale => en
 timezone => 欧洲/巴黎
 match => [grepdate，yyyyMMdd] 
 target => grepdate
} 
  

正则表达式

  DATELINE（=日期：（？< mydate> [0-9] {8}））
  

我注意到， grepdate 字段正确添加到所有事件中 - 这是我想要的 - 但是该字段的值不是日期本身（％{mydate} 的值），而是实际的字符串 ％{mydate}，除了第一次实际匹配时（解析日志文件中的实际日期， grepdate 字段包含正确的值）

我可以如何解决这个问题？

非常感谢任何帮助。

编辑：

我正在尝试一个解决方案，其中包括使用 memorize 插件。但是，我收到以下错误：

由于以下插件
不起作用，因此无法使用1个以上的过滤器工作有多个工作者：记住

有没有办法让这个过滤器线程安全？

解决方案

也许你应该使用官方的记住不是官方的，因此可以为aggregate.htmlrel =nofollow> 聚合过滤器而不能与Logstash> 2.0 一起使用。

它将如下所示：

 ＃与现在相同
 grok {
 patterns_dir => [...]
 match => [message，％{DATELINE}] 
 tag_on_failure => [not_date_line] 
 
} 
＃添加一个虚构的taskId字段来关联所有行
 mutate {
 add_field => {taskId=> all} 
} 
 
＃如果我们正在处理第一行，记住日期
如果not_date_line不在[tags] {
 aggregate { 
 task_id => ％{taskId}
 code => map ['mydate'] = event ['mydate']
} 
} 
＃如果我们正在处理下一行，添加日期
 else {
 aggregate {
 task_id => ％{taskId}
 code => event ['mydate'] = map ['mydate']
 map_action => update
 timeout => 0 
} 
} 
  

所有的事件都会有一个 mydate 字段与第一个日志行上的日期。

I have a date that is only present once in every log file and I am trying to add this date to all following events after it has been matched once, making it act like a global variable in some ways. (The date is at the top of the document and I am unable to use multiline or make changes to the file name or content)

For this, my approach is to use a grep filter with drop => false.

grok {
    patterns_dir => "[...]"
    match => [ "message", "%{DATELINE}" ]
    tag_on_failure => [ ]
}
grep {
    add_field => { "grepdate" => "%{mydate}" }
    drop => false
}
date {
    locale => "en"
    timezone => "Europe/Paris"
    match => [ "grepdate", "yyyyMMdd" ]
    target => "grepdate"
}

Regular expression:

DATELINE (= Date: (?<mydate>[0-9]{8}))

What I notice is that the grepdate field is correctly being added to all events - which is what I want - but the value of that field is not the date itself (the value of %{mydate}), but the actual string "%{mydate}", except when actually being matched for the first time (when parsing the actual date in my log file, the grepdate field contains the correct value)

What can I do to fix this?

Any help is greatly appreciated.

Edit:

I am now trying a solution that includes the use of the memorizeplugin. However, I am getting the following error:

Cannot use more than 1 filter worker because the following plugins don't work with more than one worker: memorize

Is there a way to make this filter thread-safe?

解决方案

Maybe you should use the official aggregate filter for this, since memorize is not official and will not work with Logstash >2.0.

It would go like this:

# same as what you have now
grok {
    patterns_dir => "[...]"
    match => [ "message", "%{DATELINE}" ]
    tag_on_failure => [ "not_date_line" ]

}
# add a fictional taskId field to correlate all lines
mutate {
   add_field => { "taskId" => "all" }
}

# if we're processing the first line, remember the date
if "not_date_line" not in [tags] {
    aggregate {
        task_id => "%{taskId}"
        code => "map['mydate'] = event['mydate']"
    }
} 
# if we're processing the next lines, add the date
else {
    aggregate {
        task_id => "%{taskId}"
        code => "event['mydate'] = map['mydate']"
        map_action => "update"
        timeout => 0
    }
}

All your events will then have a mydate field with the date that was on the first log line.

这篇关于Logstash：在事件之间保持一个值的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！