在Logstash中检索RESTful GET参数 [英] Retrieving RESTful GET parameters in logstash

问题描述

我正在尝试获取logstash来解析来自我的ELB日志文件的HTTP get请求中的键/值对.

I am trying to get logstash to parse key-value pairs in an HTTP get request from my ELB log files.

请求字段看起来像 http://aaa.bbb/get?a=1&b=2

我希望上面的日志行中有一个a和b的字段，但我很难弄清楚.

I'd like there to be a field for a and b in the log line above, and I am having trouble figuring it out.

我的logstash conf(为清晰起见而设置)低于其下面，不会加载任何其他关键字段.我假设我需要拆分URI的地址部分，但尚未弄清楚.

My logstash conf (formatted for clarity) is below which does not load any additional key fields. I assume that I need to split off the address portion of the URI, but have not figured that out.

input {
    file {
        path => "/home/ubuntu/logs/**/*.log"
        type => "elb"
        start_position => "beginning"
        sincedb_path => "log_sincedb"
    }
}
filter {
    if [type] == "elb" {
        grok {
            match => [ "message", "%{TIMESTAMP_ISO8601:timestamp} 
%{NOTSPACE:loadbalancer} %{IP:client_ip}:%{NUMBER:client_port:int}
%{IP:backend_ip}:%{NUMBER:backend_port:int} 
%{NUMBER:request_processing_time:float}
%{NUMBER:backend_processing_time:float} 
%{NUMBER:response_processing_time:float} 
%{NUMBER:elb_status_code:int}
%{NUMBER:backend_status_code:int} 
%{NUMBER:received_bytes:int} %{NUMBER:sent_bytes:int} 
%{QS:request}" ]
        }
        date {
            match => [ "timestamp", "ISO8601" ]
        }
        kv {
            field_split => "&?"
            source => "request"
            exclude_keys => ["callback"]
        }
    }
}


output {
    elasticsearch { host => localhost }
}

推荐答案

kv will take a URL and split out the params. This config works:

input {
    stdin { }
}

filter {
    mutate {
            add_field => { "request" => "http://aaa.bbb/get?a=1&b=2" }
    }

    kv {
            field_split => "&?"
            source => "request"
    }
}

output {
    stdout {
            codec => rubydebug
    }
}

stdout显示:

{
   "request" => "http://aaa.bbb/get?a=1&b=2",
         "a" => "1",
         "b" => "2"
}

也就是说，我鼓励您创建自己的默认URI模式版本，以便它们设置字段.然后，您可以将querystring字段传递给kv.这样更干净.

That said, I would encourage you to create your own versions of the default URI patterns so that they set fields. You can then pass the querystring field off to kv. It's cleaner that way.

更新:

对于制作自己的模式"，我的意思是采用现有的模式并根据需要对其进行修改.在logstash 1.4中，安装它们就像将它们放在新文件"patterns"目录中一样容易.我还不知道> 1.4的模式.

For "make your own patterns", I meant to take the existing ones and modify them as needed. In logstash 1.4, installing them was as easy as putting them in a new file the 'patterns' directory; I don't know about patterns for >1.4 yet.

MY_URIPATHPARAM %{URIPATH}(?:%{URIPARAM:myuriparams})?
MY_URI %{URIPROTO}://(?:%{USER}(?::[^@]*)?@)?(?:%{URIHOST})?(?:%{MY_URIPATHPARAM})?

然后，您可以在grok {}模式中使用MY_URI，它将创建一个名为myuriparams的字段，您可以将其馈送到kv {}.

Then you could use MY_URI in your grok{} pattern and it would create a field called myuriparams that you could feed to kv{}.

这篇关于在Logstash中检索RESTful GET参数的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！