如何根据日期在logstash中删除给定事件? [英] How to drop a given event in logstash based on date?

问题描述

我有一个Kafka JSON流，我将其设置为Logstash中的输入.

I have a Kafka JSON stream that I set as my input in logstash.

我想删除日期早于给定日期(例如今天的午夜)的事件.

I would like to drop events for which dates are before a given date (say, today's midnight).

我可以正确解析输入(它是json，因此是默认设置)，并且可以使用json编解码器将其打印到stdout.

I can parse the input correctly (it's json, so it's default), and I can print it to stdout with the json codec.

如何过滤日期?是否有类似的东西:

How do I filter the date? Is there something like:

filter {
  if [date] <= "some date" {
    drop { }
  }
}

推荐答案

在过滤日期{}之前，放入ruby {}并隐藏服务器的当前时间:

Before your date{} filter, drop into ruby{} and tuck away the server's current time:

event['server_timestamp'] = event['@timestamp']

然后像往常一样使用日期{}过滤器将@timestamp重置为事件的时间.

Then use your date{} filter as normal to reset @timestamp to the event's time.

之后，再次放入ruby {}以计算差异:

After that, drop into ruby{} again to compute the difference:

event['lag'] = ( ( event['server_timestamp'] - event['@timestamp'] ) ).to_f

并返回logstash，对照您的约束检查延迟:

And back in logstash, check the lag against your constraints:

# seconds!
if [lag] > 60 {
     drop{}
}

如果您不想与服务器时间进行比较，则可以使用任何时间.当我尝试使用ruby的Datetime时，它似乎减少了毫秒，因此请注意.

If you don't want to compare to the server's time, you can use any time you want. When I tried to use ruby's Datetime, it seemed to drop milliseconds, so beware of that.

这篇关于如何根据日期在logstash中删除给定事件?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！