来自json解析器的Logstash输出未发送到弹性搜索 [英] Logstash output from json parser not being sent to elasticsearch
问题描述
这是我从另一个问题的一个跟进：
JSON解析器在logstash忽略数据？
但是这次我觉得问题比上次更清楚，可能更容易让人回答。
我正在使用JSON解析器：
json #Parse所有的JSON
{
source => MFD_JSON
target => PARSED
add_field => {％{FAMILY_ID}=> ％{[PARSED] [platform] [family_id] [1]} _％{[PARSED] [platform] [family_id] [0]}}
}
logstash.stdout中的一个日志的输出部分如下所示：
FACILITY_NUM=> 1，
LEVEL_NUM=> 7，
PROGRAM=> mfd_status，
TIMESTAMP=> $ 1 $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ $ are are are are are are are are are are are are are are are are are are are are are are are are are are are当我删除JSON代码时，一堆类似上述的字段工作。当我添加JSON过滤器时，由于某些原因，整个日志就会从弹性服务器/ kibana中消失。 JSON过滤器添加的位是： PARSED=> {
platform=> {
boot_mode=> [
[0] 2，
[1]NAND
]，
boot_ver=> [
[0] 6，
[1] 1，
[2] 32576，
[3] 0
]，
全部其他VARIABLES
family_id=> [
[0] 14，
[1]Hatchetfish
]，
A全部更多变量
}，
flash=> ; [
[0] 131072，
[1] 7634944
]，
can_id=> 1700，
version=> {
kernel=> 3.0.35＃2 SMP PREEMPT Thu Aug 20 10:40:42 UTC 2015，
platform=> 17.0.32576-r1，
product=> next，
app=> 53.1.9，
boot=> 2013.04（2015年8月20日 - 10:33:51）
}
}，
％{FAMILY_ID}=> Hatchetfish 14
假装JSON不起作用，现在我还好，这不应该与popse / kibana的日志一起搞乱。另外，最后我将FAMILY_ID作为一个字段，我使用add_field单独添加。至少应该显示出来，对吗？
如果有人看到这样的东西，那将是很大的帮助。
也很抱歉垃圾邮件几乎相同的问题两次。
样本日志行：
1452470936.88 1448975468.00 1 7 mfd_status 000E91DCB5A2 load {up：[38,1.66,0.40,0.13]，mem：[967364,584900,3596,116772]，cpu：[1299,812， 1791,3157,480,144]，cpu_dvfs：[996,1589,792,871,396,1320]，cpu_op：[996,50]}
样本行将被解析（加载后的所有内容都是JSON），而在stdout中，我可以看到它被成功解析，但是我没有在弹性搜索中看到它。
这是我的输出代码：
elasticsearch
{
hosts => [localhost：9200]
document_id => ％{fingerprint}
}
stdout {codec => rubydebug}
我的另一个问题是我的logstash过滤器，但我觉得像所有相关的现在这个问题在这个问题上。
如果你想查看这里的链接： JSON解析器在logstash中忽略数据？
解决方案在这里回答我自己的问题。这不是理想的答案，但是如果有人和我有类似的问题，你可以尝试一下。
json #Parse all JSON
{
source => MFD_JSON
target => PARSED
add_field => {％{FAMILY_ID}=> ％{[PARSED] [platform] [family_id] [1]} _％{[PARSED] [platform] [family_id] [0]}}
}
这就是我之前解析了所有的JSON，我一直在尝试和错误，希望我能得到它。我即将使用Grok过滤器来获取我想要的位，这是一个选项，如果这不适合你。我稍后再回来，想到如果我把所有的东西都删除了，因为我已经忘记了一些疯狂的原因。最后我这样做：
json
{
source => MFD_JSON
target => PARSED_JSON
add_field => {FAMILY_ID=> ％{[PARSED_JSON] [platform] [family_id] [1]} _％{[PARSED_JSON] [platform] [family_id] [0]}}
remove_field => [PARSED_JSON]
}
因此，提取您感兴趣的字段/字段，然后在结束时删除由解析器创建的字段。这对我有用。我不知道为什么，但也可能适用于其他人。
This is kind of a follow up from another one of my questions:
JSON parser in logstash ignoring data?
But this time I feel like the problem is more clear then last time and might be easier for someone to answer.
I'm using the JSON parser like this:
json #Parse all the JSON
{
source => "MFD_JSON"
target => "PARSED"
add_field => { "%{FAMILY_ID}" => "%{[PARSED][platform][family_id][1]}_%{[PARSED][platform][family_id][0]}" }
}
The part of the output for one the logs in logstash.stdout looks like this:
"FACILITY_NUM" => "1",
"LEVEL_NUM" => "7",
"PROGRAM" => "mfd_status",
"TIMESTAMP" => "2016-01-12T11:00:44.570Z",
MORE FIELDS
There are a whole bunch of fields that like the ones above that work when I remove the JSON code. When I add the JSON filter, the whole log just disappears form elasticserach/kibana for some reason. The bit added by the JSON filter is bellow:
"PARSED" => {
"platform" => {
"boot_mode" => [
[0] 2,
[1] "NAND"
],
"boot_ver" => [
[0] 6,
[1] 1,
[2] 32576,
[3] 0
],
WHOLE LOT OF OTHER VARIABLES
"family_id" => [
[0] 14,
[1] "Hatchetfish"
],
A WHOLE LOT MORE VARIABLES
},
"flash" => [
[0] 131072,
[1] 7634944
],
"can_id" => 1700,
"version" => {
"kernel" => "3.0.35 #2 SMP PREEMPT Thu Aug 20 10:40:42 UTC 2015",
"platform" => "17.0.32576-r1",
"product" => "next",
"app" => "53.1.9",
"boot" => "2013.04 (Aug 20 2015 - 10:33:51)"
}
},
"%{FAMILY_ID}" => "Hatchetfish 14"
Lets pretend the JSON won't work, I'm okay with that now, that shouldn't mess with everything else to do with the log from elasticsearch/kibana. Also, at the end I've got FAMILY_ID as a field that I added separately using add_field. At the very least that should show up, right?
If someone's seen something like this before it would be great help.
Also sorry for spamming almost the same question twice.
SAMPLE LOG LINE:
1452470936.88 1448975468.00 1 7 mfd_status 000E91DCB5A2 load {"up":[38,1.66,0.40,0.13],"mem":[967364,584900,3596,116772],"cpu":[1299,812,1791,3157,480,144],"cpu_dvfs":[996,1589,792,871,396,1320],"cpu_op":[996,50]}
The sample line will be parsed (Everything after load is JSON), and in stdout I can see that it is parsed successfully, But I don't see it in elasticsearch.
This is my output code:
elasticsearch
{
hosts => ["localhost:9200"]
document_id => "%{fingerprint}"
}
stdout { codec => rubydebug }
A lot of my logstash filter is in the other question, but I think like all the relevant parts are in this question now.
If you want to check it out here's the link: JSON parser in logstash ignoring data?
解决方案 Answering my own question here. It's not the ideal answer, but if anyone has a similar problem as me you can try this out.
json #Parse all the JSON
{
source => "MFD_JSON"
target => "PARSED"
add_field => { "%{FAMILY_ID}" => "%{[PARSED][platform][family_id][1]}_%{[PARSED][platform][family_id][0]}" }
}
That's how I parsed all the JSON before, I kept at the trial and error hoping I'd get it sometime. I was about to just use a grok filter to get bits that I wanted, which is a option if this doesn't work for you. I came back to this later, and thought "What if I removed everything after" because of some crazy reason that I've forgotten. In the end I did this:
json
{
source => "MFD_JSON"
target => "PARSED_JSON"
add_field => { "FAMILY_ID" => "%{[PARSED_JSON][platform][family_id][1]}_%{[PARSED_JSON][platform][family_id][0]}" }
remove_field => [ "PARSED_JSON" ]
}
So, extract the field/fields your interested in, and then remove the field made by the parser at the end. That's what worked for me. I don't know why, but it might work for other people too.
这篇关于来自json解析器的Logstash输出未发送到弹性搜索的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
