Logstash索引JSON数组 [英] Logstash indexing JSON arrays

问题描述

Logstash很棒.我可以像这样发送JSON(多行以提高可读性):

Logstash is awesome. I can send it JSON like this (multi-lined for readability):

{
  "a": "one"
  "b": {
    "alpha":"awesome"
  }
}

，然后使用搜索词b.alpha:awesome在kibana中查询该行.很好.

And then query for that line in kibana using the search term b.alpha:awesome. Nice.

但是我现在有一个这样的JSON日志行:

However I now have a JSON log line like this:

{
  "different":[
    {
      "this": "one",
      "that": "uno"
    },
    {
      "this": "two"
    }
  ]
}

而且我希望能够通过different.this:two(或different.this:one或different.that:uno)之类的搜索找到这一行

And I'd like to be able to find this line with a search like different.this:two (or different.this:one, or different.that:uno)

如果我直接使用Lucene，我将遍历different数组，并为其中的每个哈希生成一个新的搜索索引，但是Logstash当前似乎像这样摄取该行:

If I was using Lucene directly I'd iterate through the different array, and generate a new search index for each hash within it, but Logstash currently seems to ingest that line like this:

不同:{this:一个，那个:uno}，{this:两个}

different: {this: one, that: uno}, {this: two}

这不会帮助我使用different.this或different.that搜索日志行.

Which isn't going to help me searching for log lines using different.this or different.that.

我对启用此功能的编解码器，过滤器或代码更改有任何想法吗?

Any got any thoughts as to a codec, filter or code change I can make to enable this?

推荐答案

您可以编写自己的过滤器(复制并粘贴，重命名类名称，config_name并重写filter(event)方法)或修改当前的

You can write your own filter (copy & paste, rename the class name, the config_name and rewrite the filter(event) method) or modify the current JSON filter (source on Github)

您可以在名为json.rb的以下路径logstash-1.x.x\lib\logstash\filters中找到JSON过滤器(Ruby类)源代码. JSON过滤器将内容解析为JSON，如下所示:

You can find the JSON filter (Ruby class) source code in the following path logstash-1.x.x\lib\logstash\filters named as json.rb. The JSON filter parse the content as JSON as follows

begin
  # TODO(sissel): Note, this will not successfully handle json lists
  # like your text is '[ 1,2,3 ]' JSON.parse gives you an array (correctly)
  # which won't merge into a hash. If someone needs this, we can fix it
  # later.
  dest.merge!(JSON.parse(source))

  # If no target, we target the root of the event object. This can allow
  # you to overwrite @timestamp. If so, let's parse it as a timestamp!
  if !@target && event[TIMESTAMP].is_a?(String)
    # This is a hack to help folks who are mucking with @timestamp during
    # their json filter. You aren't supposed to do anything with
    # "@timestamp" outside of the date filter, but nobody listens... ;)
    event[TIMESTAMP] = Time.parse(event[TIMESTAMP]).utc
  end

  filter_matched(event)
rescue => e
  event.tag("_jsonparsefailure")
  @logger.warn("Trouble parsing json", :source => @source,
               :raw => event[@source], :exception => e)
  return
end

您可以修改解析过程以修改原始JSON

You can modify the parsing procedure to modify the original JSON

  json  = JSON.parse(source)
  if json.is_a?(Hash)
    json.each do |key, value| 
        if value.is_a?(Array)
            value.each_with_index do |object, index|
                #modify as you need
                object["index"]=index
            end
        end
    end
  end
  #save modified json
  ......
  dest.merge!(json)

然后，您可以修改配置文件以使用/您的新的/修改后的JSON过滤器并将其放置在\logstash-1.x.x\lib\logstash\config

then you can modify your config file to use the/your new/modified JSON filter and place in \logstash-1.x.x\lib\logstash\config

这是我的elastic_with_json.conf，带有修改后的json.rb过滤器

This is mine elastic_with_json.conf with a modified json.rb filter

input{
    stdin{

    }
}filter{
    json{
        source => "message"
    }
}output{
    elasticsearch{
        host=>localhost
    }stdout{

    }
}

如果要使用新的过滤器，可以使用config_name

if you want to use your new filter you can configure it with the config_name

class LogStash::Filters::Json_index < LogStash::Filters::Base

  config_name "json_index"
  milestone 2
  ....
end

并对其进行配置

input{
    stdin{

    }
}filter{
    json_index{
        source => "message"
    }
}output{
    elasticsearch{
        host=>localhost
    }stdout{

    }
}

希望这会有所帮助.

这篇关于Logstash索引JSON数组的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！