JSON解析器在logstash中忽略数据？ [英] JSON parser in logstash ignoring data?

问题描述

我已经在这个时间了，我觉得在logstash中的JSON过滤器正在为我删除数据。我最初按照 https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-ubuntu-14-04

我已经做了一些修改，但大致相同。我的grok过滤器如下所示：

  uuid #uuid和指纹以避免重复
 {
 target = > @uuid
 overwrite =>真
} 
指纹
 {
 key => 78787878
 concatenate_sources => true 
} 
 grok＃从日志名称获取设备名称
 {
 match => {source=> ％{GREEDYDATA}％{IPV4：DEVICENAME}％{GREEDYDATA}} 
} 
 
 grok＃从日志中获取所有其他数据
 {
 match => {message=> ％{NUMBER：unixTime} ...} 
} 
 date＃将unix时间设置为适当的时间。 
 {
 match => [unixTime，UNIX] 
 target => TIMESTAMP
} 
 
 
 grok #Split该消息，如果它可以
 {
 match => {MSG_FULL=> ％{WORD：MSG_START}％{SPACE}％{GREEDYDATA：MSG_END}} 
} 
 json 
 {
 source => MSG_END
 target => JSON
} 
  

所以造成问题的位是底部，我想。我的gork东西应该是正确的。当我运行这个配置，我看到kibana中的所有内容正确显示，除了所有的日志将有JSON代码（不是所有的日志都有JSON）。当我运行它没有JSON过滤器，它显示所有。
我试图使用IF语句，以便它只运行JSON过滤器，如果它包含JSON代码，但这并没有解决任何问题。

但是，当我添加一个IF语句以仅运行特定的JSON格式（所以，如果MSG_START = x，y或z，则MSG_END将具有不同的json格式。在这种情况下，我说我只是解析z格式），然后在kibana我会看到包含x和y JSON格式的所有日志（不解析），但它不会显示z。所以我确定它必须与我使用JSON过滤器有关。

此外，每当我想用新的数据测试我开始清除旧的数据在弹性搜索，所以如果它的工作，我知道这是我的logstash的工作，而不只是从弹性搜索的内存的运行。我使用 XDELETE'http：// localhost：9200 / logstash - * /'完成此操作。但是，logstash不会在弹性搜索中创建新的索引，除非我提供文件捕获新的日志。我不知道这是否是另一个问题，只是想我应该提到它。

我希望这一切都有道理。

编辑：我只是检查logstash.stdout文件，原来是解析json，但它只是在kibana中显示_jsonparsefailure的东西，所以一定要Elastisearch错了。也许。我不知道，只是头脑风暴：）

样本日志：

1452470936.88 1448975468.00 1 7 mfd_status 000E91DCB5A2 load {up：[38,1.66,0.40,0.13]，mem：[967364,584900,3596,116772]，cpu：[1299,812,1791,3157,480,144 ]，cpu_dvfs：[996,1589,792,871,396,1320]，cpu_op：[996,50]}

MSG_START正在加载，MSG_END是上述示例中的所有内容，所以MSG_END是我要解析的有效JSON。

日志下面没有JSON，但是我的logstash将尝试解析Inf：之后的所有内容，并发出_jsonparsefailure。

1452470931.56 1448975463.00 1 6 rc.app 02:11 ：03.301 Inf：NOSApp：UpdateSplashScreen在此平台上未实现

这也是我在logstash中的输出，因为我觉得这很重要现在：

  elasticsearch 
 {
 hosts => [localhost：9200] 
 document_id => ％{fingerprint}
} 
 stdout {codec => rubydebug} 
  

解决方案

我问这个问题：来自json解析器的Logstash输出未发送到弹性搜索
以后，它有更多的相关信息，也许更好的答案，如果有人有类似的问题，我可以查看该链接。

I've been at this a while now, and I feel like the JSON filter in logstash is removing data for me. I originally followed the tutorial from https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-ubuntu-14-04

I've made some changes, but it's mostly the same. My grok filter looks like this:

uuid #uuid and fingerprint to avoid duplicates
{
    target => "@uuid"
    overwrite => true
}
fingerprint
{
    key => "78787878"
    concatenate_sources => true
}
grok #Get device name from the name of the log
{
    match => { "source" => "%{GREEDYDATA}%{IPV4:DEVICENAME}%{GREEDYDATA}" }
}

grok  #get all the other data from the log
{
    match => { "message" => "%{NUMBER:unixTime}..." }
}
date #Set the unix times to proper times.
{
    match => [ "unixTime","UNIX" ]
    target => "TIMESTAMP"
}


grok #Split up the message if it can
{
    match => { "MSG_FULL" => "%{WORD:MSG_START}%{SPACE}%{GREEDYDATA:MSG_END}" }
}
json 
{
    source => "MSG_END"
    target => "JSON"
}

So the bit causing problems is the bottom, I think. My gork stuff should all be correct. When I run this config, I see everything in kibana displayed correctly, except for all the logs which would have JSON code in them (not all of the logs have JSON). When I run it again without the JSON filter it displays everything. I've tried to use a IF statement so that it only runs the JSON filter if it contains JSON code, but that didn't solve anything.

However, when I added a IF statement to only run a specific JSON format (So, if MSG_START = x, y or z then MSG_END will have a different json format. In this case lets say I'm only parsing the z format), then in kibana I would see all the logs that contain x and y JSON format (not parsed though), but it won't show z. So i'm sure it must be something to do with how I'm using the JSON filter.

Also, whenever I want to test with new data I started clearing old data in elasticsearch so that if it works I know it's my logstash that's working and not just running of memory from elasticsearch. I've done this using XDELETE 'http://localhost:9200/logstash-*/'. But logstash won't make new indexes in elasticsearch unless I provide filebeat with new logs. I don't know if this is another problem or not, just thought I should mention it.

I hope that all makes sense.

EDIT: I just check the logstash.stdout file, it turns out it is parsing the json, but it's only showing things with "_jsonparsefailure" in kibana so something must be going wrong with Elastisearch. Maybe. I don't know, just brainstorming :)

SAMPLE LOGS:

1452470936.88 1448975468.00 1 7 mfd_status 000E91DCB5A2 load {"up":[38,1.66,0.40,0.13],"mem":[967364,584900,3596,116772],"cpu":[1299,812,1791,3157,480,144],"cpu_dvfs":[996,1589,792,871,396,1320],"cpu_op":[996,50]}

MSG_START is load, MSG_END is everything after in the above example, so MSG_END is valid JSON that I want to parse.

The log bellow has no JSON in it, but my logstash will try to parse everything after "Inf:" and send out a "_jsonparsefailure".

1452470931.56 1448975463.00 1 6 rc.app 02:11:03.301 Inf: NOSApp: UpdateSplashScreen not implemented on this platform

Also this is my output in logstash, since I feel like that is important now:

elasticsearch 
{ 
    hosts => ["localhost:9200"] 
    document_id => "%{fingerprint}"
}
stdout { codec => rubydebug }

解决方案

I asked this question: Logstash output from json parser not being sent to elasticsearch later on, and it has more relevant information on it, maybe a better answer if anyone ever has a similar problem to me you can check out that link.

这篇关于JSON解析器在logstash中忽略数据？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！