在Logstash中访问嵌套的JSON字段 [英] Access nested JSON Field in Logstash

问题描述

我在访问logstash(最新版本)中的嵌套JSON字段时遇到问题.

I have a Problem with accessing a nested JSON field in logstash (latest version).

我的配置文件如下:

input {
  http {
    port => 5001
    codec => "json"
  }
}

filter {
  mutate {
    add_field => {"es_index" => "%{[statements][authority][name]}"}
  }
  mutate {
    gsub => [
      "es_index", " ", "_"
    ]
  }
  mutate {
    lowercase => ["es_index"]
  }
  ruby {
    init => "
        def remove_dots hash
            new = Hash.new
            hash.each { |k,v|
                if v.is_a? Hash
                    v = remove_dots(v)
                end
                new[ k.gsub('.','_') ] = v
                if v.is_a? Array
                    v.each { |elem|
                        if elem.is_a? Hash
                            elem = remove_dots(elem)
                        end
                        new[ k.gsub('.','_') ] = elem
                    } unless v.nil?
                end
            } unless hash.nil?
            return new
        end
    "
    code => "
        event.instance_variable_set(:@data,remove_dots(event.to_hash))
    "
  }
}

output {
  stdout {
    codec => rubydebug
  }
  elasticsearch {
    hosts => "elasticsearch:9200"
    index => "golab-%{+YYYY.MM.dd}"
  }  
}

我有一个带有mutate的过滤器.我想添加一个可用作索引名称一部分的字段.当我使用此"%{[statements][authority][name]}"时，括号中的内容用作字符串.%{[statements][authority][name]}保存在es_index字段中. Logstash似乎认为这是一个字符串，但是为什么呢?

I have a filter with mutate. I want to add a field that I can use as a part of the index name. When I use this "%{[statements][authority][name]}" the content in the brackets is used as string.%{[statements][authority][name]} is saved in the es_indexfield. Logstash seems to think this is a string, but why?

我也尝试过使用以下表达式:"%{statements}".它的工作像预期的那样.字段语句中的所有内容都传递给es_index.如果我使用"%{[statements][authority]}"，则会发生奇怪的事情. es_index充满了与"%{statements}"产生的完全相同的输出.我想念什么?

I've also tried to use this expression: "%{statements}". It's working like expected. Everything in the field statements is passed to es_index. If I use "%{[statements][authority]}" strange things happen. es_index is filled with the exact same output that "%{statements}" produces. What am I missing?

使用"%{[statements][authority]}"的Logstash输出:

Logstash Output with "%{[statements][authority]}":

{
    "statements" => {
             "verb" => {
                 "id" => "http://adlnet.gov/expapi/verbs/answered",
            "display" => {
                "en-US" => "answered"
            }
        },
          "version" => "1.0.1",
        "timestamp" => "2016-07-21T07:41:18.013880+00:00",
           "object" => {
            "definition" => {
                       "name" => {
                    "en-US" => "Example Activity"
                },
                "description" => {
                    "en-US" => "Example activity description"
                }
            },
                    "id" => "http://adlnet.gov/expapi/activities/example"
        },
            "actor" => {
               "account" => {
                "homePage" => "http://example.com",
                    "name" => "xapiguy"
            },
            "objectType" => "Agent"
        },
           "stored" => "2016-07-21T07:41:18.013880+00:00",
        "authority" => {
                  "mbox" => "mailto:[e-mail]",
                  "name" => "GoLab",
            "objectType" => "Agent"
        },
               "id" => "0771b9bc-b1b8-4cb7-898e-93e8e5a9c550"
    },
            "id" => "a7e31874-780e-438a-874c-964373d219af",
      "@version" => "1",
    "@timestamp" => "2016-07-21T07:41:19.061Z",
          "host" => "172.23.0.3",
       "headers" => {
              "request_method" => "POST",
                "request_path" => "/",
                 "request_uri" => "/",
                "http_version" => "HTTP/1.1",
                   "http_host" => "logstasher:5001",
              "content_length" => "709",
        "http_accept_encoding" => "gzip, deflate",
                 "http_accept" => "*/*",
             "http_user_agent" => "python-requests/2.9.1",
             "http_connection" => "close",
                "content_type" => "application/json"
    },
      "es_index" => "{\"verb\":{\"id\":\"http://adlnet.gov/expapi/verbs/answered\",\"display\":{\"en-us\":\"answered\"}},\"version\":\"1.0.1\",\"timestamp\":\"2016-07-21t07:41:18.013880+00:00\",\"object\":{\"definition\":{\"name\":{\"en-us\":\"example_activity\"},\"description\":{\"en-us\":\"example_activity_description\"}},\"id\":\"http://adlnet.gov/expapi/activities/example\",\"objecttype\":\"activity\"},\"actor\":{\"account\":{\"homepage\":\"http://example.com\",\"name\":\"xapiguy\"},\"objecttype\":\"agent\"},\"stored\":\"2016-07-21t07:41:18.013880+00:00\",\"authority\":{\"mbox\":\"mailto:[e-mail]\",\"name\":\"golab\",\"objecttype\":\"agent\"},\"id\":\"0771b9bc-b1b8-4cb7-898e-93e8e5a9c550\"}"
}

您可以看到权限是es_index的一部分.因此，它没有被选作字段.

You can see that authority is part of es_index. So it was not chosen as a field.

非常感谢

推荐答案

我找到了解决方案.致谢jpcarey( Elasticsearch论坛)

I found a solution. Credits go to jpcarey (Elasticsearch Forum)

我必须删除codec => "json".这导致了另一个数据结构. statements现在是一个数组，而不是一个对象.所以我需要将%{[statements][authority][name]}更改为%{[statements][0][authority][name]}.那没有问题.

I had to remove codec => "json". That leads to another data structure. statements is now an array and not an object. So I needed to change %{[statements][authority][name]} to %{[statements][0][authority][name]}. That works without problems.

如果您遵循给定的链接，您还会发现我的mutate过滤器的更好实现.

If you follow the given link you'll also find an better implementation of my mutate filters.

这篇关于在Logstash中访问嵌套的JSON字段的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！