从事件逐个删除字段 [英] Remove Field from event by pattern

问题描述

所以我使用标准的ELK堆栈来分析Apache访问日志，这是非常有效的，但是我正在使用KV过滤器将URL参数作为字段进行分析，以便让我更好地查询。我的问题是，我正在分析的应用程序具有缓存清除动态生成的参数，这导致了成千上万的字段，每次出现一次。 ElasticSearch似乎有这样的严重问题，他们对我没有价值，所以我想删除它们。以下是模式的一个示例：


GET / page？rand123PQY = ABC& other_var = something
GET / page ？rand987ZDQ = DEF& other_var = something

在上面的示例中，我要删除的参数启动'rand'。目前，我的logstash.conf使用grok从访问日志中提取字段，其次是kv来提取Query字符串参数：


过滤器{
grok {
path => /var/log/apache/access.log
type => apache-access
}
kv {
field_split =>与 &？
}
}

有没有办法可以过滤匹配模式的任何字段 rand [A-Z0- 9] * = [A-Z0-9] * ？我看到的大多数例子都是按照确切名称定位字段，我不能使用。我想知道将请求字段正则化到一个新的字段中，运行KV，然后删除它。这个工作吗？

解决方案

如果您对感兴趣的字段集是已知的定义您可以为 kv过滤器目标 >，使用突变过滤器将有趣的字段移到消息的顶层，并删除该字段嵌套的键/值对。我认为这完全是你建议的。

或者你可以使用一个 ruby​​ filter ：

  filter {
 ruby​​ {
 code => 
 event.to_hash.keys.each {| k | 
如果k.start_with？（'rand'）
 event.remove（k）
 end 
} 

} 
} 
  

So I'm using a standard ELK stack to analyse Apache access logs, which is working well, but I'm looking to break out URL parameters as fields, using the KV filter, in order to allow me to write better queries.

My problem is that that app I'm analysing has 'cache-busting' dynamically generated parameters, which leads to tens of thousands of 'fields', each occurring once. ElasticSearch seems have severe trouble with this and they have no value to me, so I'd like to remove them. Below is an example of the pattern

GET /page?rand123PQY=ABC&other_var=something GET /page?rand987ZDQ=DEF&other_var=something

In the example above, the parameters I want to remove start 'rand'. Currently my logstash.conf uses grok to extract fields from the access logs, followed by kv to extract Query string parameters:

filter { grok { path => "/var/log/apache/access.log" type => "apache-access" } kv { field_split => "&?" } } Is there a way I can filter out any fields matching the pattern rand[A-Z0-9]*=[A-Z0-9]*? Most examples I've seen are targeting fields by exact name, which I cannot use. I did wonder about regexing the request field into a new field, running KV on that, then removing it. Would that work?

解决方案

If the set of fields that you are interested in is known and well-defined you could set target for the kv filter, move the interesting fields to the top level of the message with a mutate filter and delete the field with the nested key/value pairs. I think this is pretty much what you suggested at the end.

Alternatively you could use a ruby filter:

filter {
  ruby {
    code => "
      event.to_hash.keys.each { |k|
        if k.start_with?('rand')
          event.remove(k)
        end
      }
    "
  }
}

这篇关于从事件逐个删除字段的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！