在logstash中的列表中解析JSON [英] Parse json in a list in logstash

问题描述

我有一个

[
    {
        "foo":"bar"
    }
]

我正在尝试使用logstash中的json过滤器对其进行过滤.但这似乎不起作用.我发现我无法使用logstash中的json过滤器解析列表json.有人可以告诉我有关此问题的任何解决方法吗?

I am trying to filter it using the json filter in logstash. But it doesn't seem to work. I found that I can't parse list json using the json filter in logstash. Can someone please tell me about any workaround for this?

更新

我的日志

IP - - 0.000 0.000 [24/May/2015:06:51:13 +0000] *"POST /c.gif HTTP/1.1"* 200 4 * user_id=UserID&package_name=SomePackageName&model=Titanium+S202&country_code=in&android_id=AndroidID&eT=1432450271859&eTz=GMT%2B05%3A30&events=%5B%7B%22eV%22%3A%22com.olx.southasia%22%2C%22eC%22%3A%22appUpdate%22%2C%22eA%22%3A%22app_activated%22%2C%22eTz%22%3A%22GMT%2B05%3A30%22%2C%22eT%22%3A%221432386324909%22%2C%22eL%22%3A%22packageName%22%7D%5D * "-" "-" "-"

上述日志的URL解码版本为

URL decoded version of the above log is

IP - - 0.000 0.000 [24/May/2015:06:51:13  0000] *"POST /c.gif HTTP/1.1"* 200 4 * user_id=UserID&package_name=SomePackageName&model=Titanium S202&country_code=in&android_id=AndroidID&eT=1432450271859&eTz=GMT+05:30&events=[{"eV":"com.olx.southasia","eC":"appUpdate","eA":"app_activated","eTz":"GMT+05:30","eT":"1432386324909","eL":"packageName"}] * "-" "-" "-"

请在我的配置文件下面找到上述日志.

Please find below my config file for the above logs..

过滤器{

urldecode{
    field => "message"
}
 grok {
  match => ["message",'%{IP:clientip}%{GREEDYDATA} \[%{GREEDYDATA:timestamp}\] \*"%{WORD:method}%{GREEDYDATA}']
}

kv {
    field_split => "&? "
}
json{
    source=> "events"
}
geoip {
    source => "clientip"
}

}

我需要解析事件，即events=[{"eV":"com.olx.southasia","eC":"appUpdate","eA":"app_activated","eTz":"GMT+05:30","eT":"1432386324909","eL":"packageName"}]

推荐答案

我假设您在文件中包含json.是的，您不能直接使用json过滤器.您必须使用多行编解码器，然后再使用json过滤器.

I assume that you have your json in a file. You are right, you cannot use the json filter directly. You'll have to use the multiline codec and use the json filter afterwards.

以下配置适用于给定的输入.但是，您可能必须更改它才能正确分离事件.这取决于您的需求和文件的json格式.

The following config works for your given input. However, you might have to change it in order to properly separate your events. It depends on your needs and the json format of your file.

Logstash配置:

input     {   
    file     {
        codec => multiline
        {
            pattern => "^\]" # Change to separate events
            negate => true
            what => previous               
        }
        path => ["/absolute/path/to/your/json/file"]
        start_position => "beginning"
        sincedb_path => "/dev/null" # This is just for testing
    }
}

filter     {
    mutate   {
            gsub => [ "message","\[",""]
            gsub => [ "message","\n",""]
        }
    json { source => message }
}

---

更新

更新后，我想我已经找到了问题.显然，由于方括号，您会得到 jsonparsefailure .解决方法是您可以手动将其删除.在kv之后和json过滤器之前添加以下mutate过滤器:

---

UPDATE

After your update I guess I've found the problem. Apparently you get a jsonparsefailure because of the square brackets. As a workaround you could manually remove them. Add the following mutate filter after your kv and before your json filter:

mutate  {
    gsub => [ "events","\]",""]
    gsub => [ "events","\[",""]
}

---

更新2

好的，假设您输入的内容如下:

---

UPDATE 2

Alright, assuming your input looks like this:

[{"foo":"bar"},{"foo":"bar1"}]

这里有4个选项:

选项a)丑陋的gsub

另一个gsub可能是一个丑陋的解决方法:

An ugly workaround would be another gsub:

gsub => [ "event","\},\{",","]

但这会删除内部关系，所以我想您不想这样做.

But this would remove the inner relations so I guess you don't want to do that.

选项b)拆分

更好的方法可能是使用拆分过滤器:

A better approach might be to use the split filter:

split {
    field => "event"
    terminator => ","
}
mutate  {
    gsub => [ "event","\]",""]
    gsub => [ "event","\[",""]
   }
json{
    source=> "event"
}

这将生成多个事件. (第一个使用foo = bar，第二个使用foo1 = bar1.)

This would generate multiple events. (First with foo = bar and second with foo1 = bar1.)

选项c)突变拆分

您可能希望将所有值都包含在一个logstash事件中.您可以使用 mutate => split 过滤器生成一个数组，并在存在条目的情况下解析json.不幸的是，您必须为每个条目设置一个条件，因为Logstash在其配置中不支持循环.

You might want to have all the values in one logstash event. You could use the mutate => split filter to generate an array and parse the json if an entry exists. Unfortunately you will have to set a conditional for each entry because logstash doesn't support loops in its config.

mutate  {
    gsub => [ "event","\]",""]
    gsub => [ "event","\[",""]
    split => [ "event", "," ]
   }

json{
    source=> "event[0]"
    target => "result[0]"
}

if 'event[1]' {
    json{
        source=> "event[1]"
        target => "result[1]"
    }
    if 'event[2]' {
        json{
            source=> "event[2]"
            target => "result[2]"
        }
    }
    # You would have to specify more conditionals if you expect even more dictionaries
}

选项d)Ruby

根据您的评论，我试图找到一种红宝石的方法.以下工作(在您的kv过滤器之后):

According to your comment I tried to find a ruby way. Following works (after your kv filter):

mutate  {
    gsub => [ "event","\]",""]
    gsub => [ "event","\[",""]
}

ruby  {
    init => "require 'json'"
    code => "
        e = event['event'].split(',')
        ary = Array.new
        e.each do |x|
            hash = JSON.parse(x)
            hash.each do |key, value|
                ary.push( { key =>  value } )
            end
        end
        event['result'] = ary
    "
}

选项e)Ruby

在您的kv过滤器之后使用此方法(无需设置突变过滤器):

Use this approach after your kv filter (without setting a mutate filter):

ruby  {
    init => "require 'json'"
    code => "
            event['result'] = JSON.parse(event['event'])
    "
}

它将解析诸如event=[{"name":"Alex","address":"NewYork"},{"name":"David","address":"NewJersey"}]

进入:

"result" => [
    [0] {
           "name" => "Alex",
        "address" => "NewYork"
    },
    [1] {
           "name" => "David",
        "address" => "NewJersey"
    }

由于kv过滤器的行为，因此不支持空格.希望您的实际输入中没有任何内容，对吗?

Since the behavior of the kv filter this does not support whitespaces. I hope you don't have any in your real inputs, do you?

这篇关于在logstash中的列表中解析JSON的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！