忽略早于给定日期的传入logstash条目 [英] ignore incoming logstash entries that are older than a given date

问题描述

我希望Logstash在处理输入条目时，只删除早于N天的条目.

I want Logstash, when it's processing input entries, to simply drop entries that are older than N days.

我假设我将使用日期模块，并且显然使用拖放，但我不知道如何连接它们.

I assume I'll use the date module and obviously drop, but I don't know how to connect them.

推荐答案

我知道进行日期级别比较的唯一方法是通过Ruby代码.您需要date过滤器来解析时间戳(这是它自己的问题).

The only way that I know to do date level comparison is via Ruby code. You need the date filter to parse the timestamp (that's its own issue).

一旦您将日期解析为一个字段(例如event["@timestamp"])，就可以使用它来确定是否要忽略它:

Once you parse the date into a field (e.g., event["@timestamp"]), then you can use it to determine if you want to ignore it or not:

5.0:

ruby {
  code => "event.cancel if (Time.now.to_f - event.get('@timestamp').to_f) > (60 * 60 * 24 * 5)"
}

Pre-5.x:

ruby {
  code => "event.cancel if (Time.now.to_f - event['@timestamp'].to_f) > (60 * 60 * 24 * 5)"
}

在这种情况下，5是N.

此外，值得指出的是，这与Logstash恰好在运行的机器时间有关.如果不正确，则会影响日期数学.同样，如果源计算机的系统时钟错误，那么也可能是一个问题.

Also, it's worth pointing out that this is relative to the machine time where Logstash happens to be running. If it's inaccurate, then it will impact date math. Similarly, if the source machine's system clock is wrong, then it too can be a problem.

利用Alain的优点，除了仅基于它进行拖延之外，您还可以使用此存储区的滞后时间.

Drawing on Alain's good point, you could use this store the lag time, in addition to just dropping based on it.

5.0:

ruby {
  code => "event.set('lag_seconds', Time.now.to_f - event.get('@timestamp').to_f))"
}

# 5 represents the number of days to allow
if [lag_seconds] > (60 * 60 * 24 * 5) {
  drop { }
}

Pre-5.x:

ruby {
  code => "event['lag_seconds'] = Time.now.to_f - event['@timestamp'].to_f)"
}

# 5 represents the number of days to allow
if [lag_seconds] > (60 * 60 * 24 * 5) {
  drop { }
}

使用这种方法，您便可以为lag_seconds编制索引，这是一个小数，从而可以分析索引中是否存在ES或其他数据存储中的滞后.

Using this approach, you would then be indexing lag_seconds, which is a fractional amount, thereby allowing you to analyze lag in your index if this goes into ES or some other data store.

这篇关于忽略早于给定日期的传入logstash条目的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！