Logstash:将URL参数放入哈希 [英] Logstash : get URL params into hash
问题描述
我正在尝试使用Logstash和ElasticSearch监视我的Apache Web服务器活动.目前,它工作得很好,但是我需要有关我的请求字段的更具体的信息. 目前,我的logstash配置为:
I'm trying to use Logstash and ElasticSearch to monitor my Apache webserver activity. At this time, it works pretty well but I need to more specific informations about my request field. At this time my logstash configuration is :
filter {
grok { match => { "message" => "%{COMBINEDAPACHELOG}" } }
grok { match => { "request" => [ "url", "%{URIPATH:url_path}%{URIPARAM:url_params}?" ]} }
urldecode{ field => "url_path" }
mutate { gsub => ["url_params","\?","" ] }
kv {
field_split => "&"
source => "url_params"
prefix => "url_param_"
}
date { match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ] }
geoip { source => "clientip" }
useragent { source => "agent" }
}
获取基本的Apache日志:
Taking a basic apache log :
255.254.230.10 - - [11/Dec/2013:00:01:45 -0800] "GET /xampp/boreal%3A123456/status.php?pretty=true&test=boreal%3A12345 HTTP/1.1" 200 3891 "http://cadenza/xampp/navi.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0"
第一个配置的结果是:
{
"message" => "255.254.230.10 - - [11/Dec/2013:00:01:45 -0800] \"GET /xampp/boreal%3A123456/status.php?pretty=true&test=boreal:%3A12345 HTTP/1.1\" 200 3891 \"http://cadenza/xampp/navi.php\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0\"",
"@version" => "1",
"@timestamp" => "2013-12-11T08:01:45.000Z",
...
"request" => "/xampp/boreal%3A123456/status.php?pretty=true&test=boreal%3A12345",
"url_path" => "/xampp/boreal:123456/status.php",
"url_params" => "pretty=true&test=boreal%3A12345",
"url_param_pretty" => "true",
"url_param_test" => "boreal%3A12345",
...
}
而且(在理想世界中),我想对url params做出以下响应:
And (in a dream world), I would like to have this response for url params :
{
...
"request" => "/xampp/boreal%3A123456/status.php?pretty=true&test=boreal%3A12345",
"url_path" => "/xampp/boreal:123456/status.php",
"url_params" => {
"pretty" => "true",
"url_param_test" => "boreal:12345"
},
...
}
我的愿望
- url_params成为哈希数组.
- 此哈希的每个键将是参数的名称
- 每个对应的值将是urldecode值
问题
- 我需要创建自己的插件(我还不熟悉ruby)吗?
- 是否存在现有插件(我没有找到……可能搜索不正确)?
- 这是没有插件的一种方法吗?
感谢您的帮助(对不起我的英语)
Thanks for your help (and sorry for my english)
雷诺(Renaud)
解决方案:
感谢瓦尔,他找到了解决方案.我将配置更改为:
Thanks to Val, He found the solution. I changed my configuration to :
grok { match => { "request" => [ "url", "%{URIPATH:url_path}%{URIPARAM:url_params}?" ]} }
urldecode{ field => "url_path" }
mutate { gsub => ["url_params","\?","" ] }
kv {
field_split => "&"
source => "url_params"
target => "url_params_hash"
}
urldecode{ field => "url_params_hash" }
使用此解决方案,即使url_params字符串中包含&"(%26)字符,分割也是正确的.
Using this solution, even if an "&"(%26) character are into url_params string the splitting is correct.
推荐答案
您几乎可以使用kv
过滤器来正确地进行操作.您需要稍微更改其配置.
You're almost doing it right using the kv
filter. You need to change its configuration a little bit.
您还需要在url_params
的路径之后,添加另一个urldecode
过滤器
You also need to add another urldecode
filter for the url_params
just after the other one for the path
urldecode{ field => "url_path" }
urldecode{ field => "url_params" }
mutate { gsub => ["url_params","\?","" ] }
kv {
field_split => "&"
source => "url_params"
target => "url_params_hash"
}
您会得到这样的东西:
{
"message" => "255.254.230.10 - - [11/Dec/2013:00:01:45 -0800] \"GET /xampp/boreal%3A123456/status.php?pretty=true&test=boreal:%3A12345 HTTP/1.1\" 200 3891 \"http://cadenza/xampp/navi.php\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0\"",
"@version" => "1",
"@timestamp" => "2013-12-11T08:01:45.000Z",
"url_params_hash" => {
"pretty" => "true",
"test" => "boreal:12345"
}
}
这篇关于Logstash:将URL参数放入哈希的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!