我如何在Logstash 5.1.2中为Tomcat/Java配置多行 [英] How i can config multiline in logstash 5.1.2 for tomcat/java

问题描述

我使用5.1.2versón的logstash，filebeat，elasticsearch ..."ELK"

我尝试从tomcat服务器发送日志(catalina.out和apps-java日志)，但是不能，因为存在logstash多行过滤器/编解码器的配置问题.

我遵循此说明
https://blog.lanyonm. org/articles/2014/01/12/logstash-multiline-tomcat-log-parsing.html

Logstash.conf是这样的:

input {
    beats {
    port => 9000
    }
}

filter {
  if [type] == "tomcat-pro" {
    codec => "multiline" {
      patterns_dir => "/opt/logstash/patterns"
      pattern => "(^%{TOMCAT_DATESTAMP})|(^%{CATALINA_DATESTAMP})"
      negate => true
      what => "previous"
    }
  }
}

output {
    elasticsearch {
        hosts => ["localhost:9200"]
        index => "tomcat-pro"
    }   
}

Logstash接收filebeat文件.
Filebeat.yml

filebeat.prospectors:
- input_type: log
  document_type: tomcat-pro
  paths:
  - /opt/tomcat-test/logs/catalina.out

当我启动该服务时，控制台会向我显示以下内容:

[2017-01-26T13:10:33,712][ERROR][logstash.agent           ] fetched an invalid config {:config=>"input {\n    beats {\n    port => 9000\n    }\n}\n\nfilter {\n  if [type] == \"tomcat-pro\" {\n    codec => \"multiline\" {\n      patterns_dir => \"/opt/logstash/patterns\"\n      pattern => \"(^%{TOMCAT_DATESTAMP})|(^%{CATALINA_DATESTAMP})\"\n      negate => true\n      what => \"previous\"\n    }\n  }\n}\n\noutput {\n    elasticsearch {\n        hosts => [\"localhost:9200\"]\n        index => \"tomcat-pro\"\n    }   \n}\n", :reason=>"Expected one of #, { at line 9, column 11 (byte 96) after filter {\n  if [type] == \"tomcat-pro\" {\n    codec "}

摘要:

fetched an invalid config
reason=>"Expected one of #, { at line 9, column 11 (byte 96) after filter {\n  if [type] == \"tomcat-pro\" {\n    codec "}

我在Google中读过，建议在文件格式中使用多行而不是在logstash中使用多行，但是我没有很好地配置...

有人可以帮助我吗? :(

PD:我是西班牙语，对不起，谷歌翻译". Si puedes响应者，西班牙人，很大程度;)

解决方案

我认为在测试过，因此请针对实际日志.

filebeat.prospectors:
- document_type: catalina-wine-mixer
  paths:
  - /opt/tomcat-test/logs/catalina.out
  multiline.pattern: '^([0-9]{4}-[0-9]{2}-[0-9]{2})|([J|F|M|A|M|S|O|N|D][a-z]{2} [0-9]{1,2}, [0-9]{2})'
  multiline.negate: true
  multiline.match: after

output.elasticsearch:
  hosts: ['http://localhost:9200']

I use a 5.1.2 verisón of logstash, filebeat, elasticsearch... "ELK"

I try send logs from tomcat server (catalina.out and apps-java logs) but can´t because have problems of config of logstash multiline filter/codec.

I follow this instructions
https://blog.lanyonm.org/articles/2014/01/12/logstash-multiline-tomcat-log-parsing.html

Logstash.conf is this:

input {
    beats {
    port => 9000
    }
}

filter {
  if [type] == "tomcat-pro" {
    codec => "multiline" {
      patterns_dir => "/opt/logstash/patterns"
      pattern => "(^%{TOMCAT_DATESTAMP})|(^%{CATALINA_DATESTAMP})"
      negate => true
      what => "previous"
    }
  }
}

output {
    elasticsearch {
        hosts => ["localhost:9200"]
        index => "tomcat-pro"
    }   
}

Logstash receives files of filebeat.
Filebeat.yml

filebeat.prospectors:
- input_type: log
  document_type: tomcat-pro
  paths:
  - /opt/tomcat-test/logs/catalina.out

When i start the service the console show me this:

[2017-01-26T13:10:33,712][ERROR][logstash.agent           ] fetched an invalid config {:config=>"input {\n    beats {\n    port => 9000\n    }\n}\n\nfilter {\n  if [type] == \"tomcat-pro\" {\n    codec => \"multiline\" {\n      patterns_dir => \"/opt/logstash/patterns\"\n      pattern => \"(^%{TOMCAT_DATESTAMP})|(^%{CATALINA_DATESTAMP})\"\n      negate => true\n      what => \"previous\"\n    }\n  }\n}\n\noutput {\n    elasticsearch {\n        hosts => [\"localhost:9200\"]\n        index => \"tomcat-pro\"\n    }   \n}\n", :reason=>"Expected one of #, { at line 9, column 11 (byte 96) after filter {\n  if [type] == \"tomcat-pro\" {\n    codec "}

Summary:

fetched an invalid config
reason=>"Expected one of #, { at line 9, column 11 (byte 96) after filter {\n  if [type] == \"tomcat-pro\" {\n    codec "}

I read in google that is recommended to use multiline in filebeat rather than in logstash, but i dont config very well...

Someone can help me? :(

PD: Im spanish, sorry for "google translate". Si puedes responder en español, sería mucho mejor ;)

解决方案

I think doing the multiline processing in Filebeat is the way to go, so instead of debugging the Logstash configuration error you posted I will show a Filebeat configuration where Filebeat combines the lines before shipping the event.

If you were only using Logstash for the multiline filter, then you could just output directly to Elasticsearch from Filebeat. But if you do need to output to Logstash please follow the instructions for configuring Filebeat to be used with Logstash.

The pattern I use below hasn't been thoroughly tested so please test it against the actual logs.

filebeat.prospectors:
- document_type: catalina-wine-mixer
  paths:
  - /opt/tomcat-test/logs/catalina.out
  multiline.pattern: '^([0-9]{4}-[0-9]{2}-[0-9]{2})|([J|F|M|A|M|S|O|N|D][a-z]{2} [0-9]{1,2}, [0-9]{2})'
  multiline.negate: true
  multiline.match: after

output.elasticsearch:
  hosts: ['http://localhost:9200']

这篇关于我如何在Logstash 5.1.2中为Tomcat/Java配置多行的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！