解析流利的nginx入口日志 [英] Parse nginx ingress logs in fluentd

问题描述

我想使用Kubernetes中的fluentd解析入口nginx日志.在Logstash中，这非常容易，但是我对流利的语法感到困惑.

I'd like to parse ingress nginx logs using fluentd in Kubernetes. That was quite easy in Logstash, but I'm confused regarding fluentd syntax.

现在我有以下规则:

<source>
  type tail
  path /var/log/containers/*.log
  pos_file /var/log/es-containers.log.pos
  time_format %Y-%m-%dT%H:%M:%S.%NZ
  tag kubernetes.*
  format json
  read_from_head true
  keep_time_key true
</source>

<filter kubernetes.**>
  type kubernetes_metadata
</filter>

结果是我得到了这个日志，但是它没有被解析:

And as a result I get this log but it is unparsed:

127.0.0.1 - [127.0.0.1] - user [27/Sep/2016:18:35:23 +0000] "POST /elasticsearch/_msearch?timeout=0&ignore_unavailable=true&preference=1475000747571 HTTP/2.0" 200 37593 "http://localhost/app/kibana" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Centos Chromium/52.0.2743.116 Chrome/52.0.2743.116 Safari/537.36" 951 0.408 10.64.92.20:5601 37377 0.407 200

我想应用过滤规则，以便能够通过Kibana中的IP地址，HTTP方法等进行搜索.我该如何实施?

I'd like to apply filter rules to be able to search by IP address, HTTP method, etc in Kibana. How can I implement that?

推荐答案

管道在logstash和fluentd中有很大的不同.并且花了一些时间来构建有效的Kubernetes-> Fluentd-> Elasticsearch-> Kibana解决方案.

Pipelines are quite different in logstash and fluentd. And it took some time to build working Kubernetes -> Fluentd -> Elasticsearch -> Kibana solution.

我的问题的简短答案是安装 fluent-plugin-parser 插件(我想知道为什么它没有在标准软件包中提供)并将此规则放在 kubernetes_metadata 之后过滤器:

Short answer to my question is to install fluent-plugin-parser plugin (I wonder why it doesn't ship within standard package) and put this rule after kubernetes_metadata filter:

<filter kubernetes.var.log.containers.nginx-ingress-controller-**.log>
  type parser
  format /^(?<host>[^ ]*) (?<domain>[^ ]*) \[(?<x_forwarded_for>[^\]]*)\] (?<server_port>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+[^\"])(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")? (?<request_length>[^ ]*) (?<request_time>[^ ]*) (?:\[(?<proxy_upstream_name>[^\]]*)\] )?(?<upstream_addr>[^ ]*) (?<upstream_response_length>[^ ]*) (?<upstream_response_time>[^ ]*) (?<upstream_status>[^ ]*)$/
  time_format %d/%b/%Y:%H:%M:%S %z
  key_name log
  types server_port:integer,code:integer,size:integer,request_length:integer,request_time:float,upstream_response_length:integer,upstream_response_time:float,upstream_status:integer
  reserve_data yes
</filter>

此处提供了很多示例的完整答案: https://github.com/kayrus/elk- kubernetes/

Long answer with lots of examples is here: https://github.com/kayrus/elk-kubernetes/

这篇关于解析流利的nginx入口日志的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！