Nginx grok模式的logstash [英] Nginx grok pattern for logstash

问题描述

以下是我的 Nginx日志格式

log_format timed_combined'$ http_x_forwarded_for-$ remote_user[$ time_local]''"$ request" $ status $ body_bytes_sent''"$ http_referer""$ http_user_agent"''$ request_time $ upstream_response_time $ pipe';

log_format timed_combined '$http_x_forwarded_for - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent" ' '$request_time $upstream_response_time $pipe';

以下是 Nginx日志条目(供参考)

- - test.user [26/May/2017:21:54:26 +0000] "POST /elasticsearch/_msearch HTTP/1.1" 200 263 "https://myserver.com/app/kibana" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36" 0.020 0.008 .

以下是 logstash grok模式

NGUSERNAME [a-zA-Z\.\@\-\+_%]+
NGUSER %{NGUSERNAME}
NGINXACCESS %{IPORHOST:clientip} - - \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{NUMBER:request_time} %{NUMBER:upstream_time}

在logstash日志中发现错误

状态" => 400，错误" => {类型" =>"mapper_parsing_exception"，"reason" =>无法解析[timestamp]"，"caused_by" => {"type" =>"illegal_argument_exception"，原因" =>无效格式:\"26/May/2017:19:28:14 -0400 \"格式错误\"/May/2017:19:28:14 -0400 \"

"status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [timestamp]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: \"26/May/2017:19:28:14 -0400\" is malformed at \"/May/2017:19:28:14 -0400\"

Issue: - Nginx logs are not getting grokked. 
Requirement: - Timestamp should be filtered into a particular field.

我的配置有什么问题?如何解决这个错误?

What's wrong in my configuration? How to fix this error?

推荐答案

您提供的日志行与默认的 NGINXACCESS grok模式不匹配，因为存在以下两个差异:

The log line you provided does not match the default NGINXACCESS grok pattern because of two differences:

1. 作为日志行中的第一个元素，应该使用IP地址或主机名，但是在日志行中，破折号(-)是第一个元素.
2. 日志行中的第三个元素是用户名，但是grok模式要求使用破折号(-)

1. As the first element in the log line an ip address or hostname is expected, but in your log line a dash (-) is the first element.
2. The third element in your log line is a username, but the grok pattern expects a dash (-)

因此，有两种方法可以解决此问题:

So there are two way to resolve this:

1. 请确保您的日志行与默认模式匹配
2. 将grok模式更改为如下形式:

NGINXACCESS - - %{USERNAME:username} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent} %{NUMBER:request_time} %{NUMBER:upstream_time}

我建议使用 Grok调试器来开发和调试grok模式.它允许您逐步创建和测试它们.

I suggest to use Grok debugger to develop and debug grok patterns. It allows you to create and test them incrementally.

这篇关于Nginx grok模式的logstash的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！