Logstash在一个日志文件中处理多种文本格式 [英] Logstash handle more than one format of text in one log file

问题描述

我正在使用ELK(filebeat，logstash，elasticsearch，kibana)进行日志管理.在一个日志文件中，我有三种格式.我正在使用

I am using ELK(filebeat,logstash,elasticsearch,kibana) to make log management. In one log file, I have three kinds of format. I am using

在一种格式中，我有date + parameters + json + stacktrace.这种格式的文本有多行. 在第二种格式中，它只是date + requestMethod(获取或发布)+一些文本.它在一行中. 在第三种格式中，它具有date +模块化名称:(在这种情况下，它是paymentAdmin)+ json

In one format, I have date+parameters+json+stacktrace. This kind of format text has multiple lines. In second format, it is just date+requestMethod(Get or post)+some text. It is in one line. In third format, it has date+ modular Name:(In this case, it is paymentAdmin)+json

我想我可以通过if，else if子句和3种匹配器来使用logStash处理这三种格式.有人可以向我展示该示例如何编写if，else if子句以在一个日志文件中处理3种格式的示例吗?

I suppose I could use logStash to handle these three kinds of format by if, else if clause, and 3 kinds of matchers. Can someone show me the example how to write if, else if clause to handle 3 formats in one log file?

2018-04-10 10:14:18 params:{"currentPage":"1","schoolname":"","schoolcode":"","pageSize":"10","pageNumber":"1","ispage":"yes","sortOrder":"asc","_":"1523326435639"} java.lang.NullPointerException: null
    at com.datalook.group.BusinessHandler.queryGroup(BusinessHandler.java:204) ~[classes/:?]
    at com.datalook.group.BusinessGroupController.businessGroup(BusinessGroupController.java:43) [classes/:?]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_77]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_77]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_77]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_77]

121.251.19.243 - - [02/Jan/2018:00:00:13 +0800] "GET /eams5-student/static/cryptoJS/sha1.js HTTP/1.1" 404 - "http://jxglstu.hfut.edu.cn/eams5-student/login" "Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; .NET4.0C; .NET4.0E; InfoPath.3; rv:11.0) like Gecko" 0.0.0.107 80 0 - "101.226.226.221" "-"


2018-03-28 13:23:01 PaymentAdmin:{"manageamt":"0.00","managefee":0.2,"pcode":"01","txamt":"0.01"}

推荐答案

有多种方法可以实现，您可以定义具有多个模式的单个grok过滤器进行匹配，诸如此类，

There are multiple ways it can be achieved, you can either define single grok filter with multiple patterns to match, something like this,

filter {
   grok {
     match => [ "message", "PATTERN1", "PATTERN2" ]
    }
}

或

if "SUCCESS" not in [tags]{
  grok {
    match => { "message" => PATTERN1}
    add_tag => ["SUCCESS"]
    remove_tag => ["_grokparsefailure"]
  }
}

if "SUCCESS" not in [tags]{
  grok {
    match => { "message" => PATTERN2}
    add_tag => ["SUCCESS"]
    remove_tag => ["_grokparsefailure"]
  }
}

或者，通过通过输入设置类型的有条件方法插件

if [type] == "syslog" {
  grok {
    match => ["message", "PATTERN1"]
  }
}

else if [type] == "nginx" {
      grok {  
        match => ["message", "PATTERN2"]
      }
  }

上述方法的问题在于，将按顺序应用模式，直到找到匹配项为止.

The problem with above approach is that the patterns will be applied in order until a match is found.

这篇关于Logstash在一个日志文件中处理多种文本格式的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！