用邮递员将ELK Beat输出模拟到Logstash [英] Mock an ELK Beat output to Logstash with Postman

问题描述

我有Elastic Beats(例如metricbeat和auditbeat)，它们可以将数据发送到logstash.为了进行测试，我想模拟一些节拍到logstash输入的数据.

我在beats(yaml配置文件)中使用logstash输出插件:

  output.logstash:已启用:true主机:["127.0.0.1:5044"] 

和logstash(.conf管道配置文件)中的beats输入插件:

 输入{节拍{端口=>"5044"}} 

我天真地认为节拍通过HTTP请求与logstash通信，所以我使用Postman向logstash发出POST请求，但是在logstash中出现了 InvalidFrameProtocolException:节拍协议的无效版本错误./p>

我试图用Wireshark检查数据包，以查看原始数据包的格式，但我所看到的只是TCP数据包，没有HTTP协议.我想那时插件不能通过HTTP请求进行通信.

具有更多专业知识的人可以对此发表评论吗?有没有一种方法可以通过邮递员将节拍输出数据模拟到logstash?

解决方案

感谢@leandrojmp，我知道节拍和logstash是通过伐木工人协议而不是HTTP进行通信的.

我最终使用了http logstash输入插件，以便可以通过邮递员模拟我的beats事件.虽然我没有修改原始的管道配置文件，但是我使用了logstash的功能来加载多个管道配置文件.因此，我用http插件和一些过滤器创建了另一个logstash管道配置文件，以删除http特定数据，例如"header".字段:

 输入{标准输入{}http {端口=>"8081"}}筛选 {变异{remove_field =>[标题"]}} 

然后，当我要启动Logstash进行测试时，可以将两个文件放在同一目录中来加载这两个文件:

  bin/logstash -f logstash.d/*.conf 

或者在我不进行测试时只是原始配置文件:

  bin/logstash -f logstash.d/pipeline.conf 

I have Elastic Beats (like metricbeat and auditbeat) that send data to logstash. For testing purposes, I'd like to mock some data from the beats to the logstash input.

I'm using the logstash output plugin in the beats (yaml config file):

output.logstash:
  enabled: true
  hosts: ["127.0.0.1:5044"]

and the beats input plugin in logstash (.conf pipeline config file):

input {
    beats {
        port => "5044"
    }
}

I naïvely assumed the beats were communicating with logstash via HTTP requests so I used Postman to make a POST request to logstash, but I got an InvalidFrameProtocolException: Invalid version of beats protocol error in logstash.

I tried to inspect the packets with wireshark to see how the original packets are formatted but all I could see was TCP packets and no HTTP protocol. I guess then that the plugins don't communicate via HTTP requests.

Can someone with more expertise comment on this ? Is there a way to mock beats output data to logstash via postman ?

解决方案

Thanks to @leandrojmp I know the beats and logstash communicate via the lumberjack protocol and not HTTP.

I ended up using the http logstash input plugin so that I can mock my beats events via postman. I didn't modify my original pipeline configuration file though, but I used the ability of logstash to load several pipeline configuration files. So I created another logstash pipeline configuration file with my http plugin and some filter to remove the http specific data like the "header" field:

input {
    stdin { }
    http {
        port => "8081"
    }
}

filter {
    mutate {
        remove_field => ["headers"]
    }
}

I can then load the two files when I want to launch logstash for testing, by placing them in the same directory:

bin/logstash -f logstash.d/*.conf

Or just the original config file when I'm not testing:

bin/logstash -f logstash.d/pipeline.conf

这篇关于用邮递员将ELK Beat输出模拟到Logstash的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！