弹性搜索重新启动后，Logstash停止写入弹性搜索？ [英] Logstash stops writting to elasticsearch after an elasticsearch restart?

问题描述

我对Logstash / Elasticsearch / Kibana来说相当新鲜，弹性搜索停止并重新启动后，恢复logstash有一个奇怪的问题。我有一个使用logstash-1.4.2和elasticsearch-1.3.4的ELK安装。

这是我的场景：

1 - 开始弹性搜索
2 - 启动logstash logforwarder on远程节点
3 - 日志文件/事件已成功写入弹性搜索
4 - 重新启动弹性搜索
5 - Logstash重新连接到弹性搜索（我看到以下消息），但事件不再插入弹性搜索

Logstash日志：

  log4j，[2014-11-15T16：16：20.261] INFO：org.elasticsearch.cluster.service：[logstash-grading-n1-76848-4038] removed {[logstash-rest-n1-168118-4018] [o6Gxxd1SQC2XchlsySaz8Q] [rest-n1] [inet [/10.165.0.31:9300]] {client = true，data = false}，[Dorma] [DoIcmge9QdqxiRnmBjtVdg] [build] [inet [/10.165.1.140:9300]]， [logstash-web-n1-221006-4016] [vRY0Ib7oTyOuliOYekP_nA] [web-n1] [inet [/10.165.0.21:9300]] {client = true，data = false}，[logstash-flume-n1-99849-4018 ] [kNknQCs0TPi6-VgiC4f-8A] [flume-n1] [inet [/10.165.0.41:9300]] {client = true，data = false}，}，原因：zen-disco-mas ter_failed（[Dorma] [DoIcmge9QdqxiRnmBjtVdg] [build] [inet [/10.165.1.140:9300]]）
 log4j，[2014-11-16T09：08：38.706] INFO：org.elasticsearch.cluster.service： [logstash-grading-n1-76848-4038] detected_master [Masque] [5mLN45_iTfq_YuFKV20OLg] [build] [inet [/10.165.1.140:9300]]，添加了{[logstash-rest-n1-168118-4018] [o6Gxxd1SQC2XchlsySaz8Q] [ rest-n1] [inet [/10.165.0.31:9300]] {client = true，data = false}，[logstash-web-n1-221006-4016] [vRY0Ib7oTyOuliOYekP_nA] [web-n1] [inet [/10.165。 0.21：9300]] {client = true，data = false}，[Masque] [5mLN45_iTfq_YuFKV20OLg] [build] [inet [/10.165.1.140:9300]]，}，原因：zen-disco-receive（from master [ Masque] [5mLN45_iTfq_YuFKV20OLg] [build] [inet [/10.165.1.140:9300]]]）
 log4j，[2014-11-16T09：08：38.719] INFO：org.elasticsearch.cluster.service：[logstash加入{[logstash-flume-n1-99849-4018] [KNknQCs0TPi6-VgiC4f-8A] [flume-n1] [inet [/10.165.0.41:9300]] {client = true， data = false}，}，原因：zen-disco-receive（from master [[Masque] [5mLN45_iTfq_YuFKV20OLg] [build] [ inet [/10.165.1.140:9300]]]）
  

弹性搜索日志：

  [2014-11-16 09：08：36,248] [INFO] [网关] [Masque]将[8]索引恢复为cluster_state 
 ... 
 [2014-11-16 09：09：39,085] [INFO] [cluster.service] [Masque]添加了{[logstash-grading-n1-76848-4038] [8szJ9egnQsaAvet6S10Tmw] [grading- n1] [inet [/10.165.0.71:9300]] {client = true，data = false}，}，reason：zen-disco-receive（join from node [[logstash-grading-n1-76848-4038] [8szJ9egnQsaAvet6S10Tmw ] [grading-n1] [inet [/10.165.0.71:9300]] {client = true，data = false}] 
  

6 - 重新启动logstash - 事件再次开始工作

所以问题是我不需要重新启动所有我的logstash 实例，因为弹性搜索重新启动。有人有什么想法吗？我缺少一些配置？

解决方案

这看起来像一个已知的logstash问题：

Elasticsearch重新启动时无限停止

https://github.com/elasticsearch/logstash/issues/1655

除了重新启动logstash之外，没有其他的工作。我会在这个问题上评论你的问题 - 越多的人谈论这个问题就越快得到解决。

I'm fairly new to Logstash/Elasticsearch/Kibana and I'm having a strange issue with recovering logstash after elasticsearch stops and restarts. I have an ELK install with logstash-1.4.2 and elasticsearch-1.3.4.

This is my scenario:

1 - Start elastic search 2 - Start logstash logforwarder on a remote node 3 - Log file/ events are successfully being written to elasticsearch 4 - Restart elasticsearch 5 - Logstash reconnects to elastic search (I see the following messages) but events are no longer being inserted to elasticsearch

Logstash log:

log4j, [2014-11-15T16:16:20.261]  INFO: org.elasticsearch.cluster.service: [logstash-grading-n1-76848-4038] removed {[logstash-rest-n1-168118-4018][o6Gxxd1SQC2XchlsySaz8Q][rest-n1][inet[/10.165.0.31:9300]]{client=true, data=false},[Dorma][DoIcmge9QdqxiRnmBjtVdg][build][inet[/10.165.1.140:9300]],[logstash-web-n1-221006-4016][vRY0Ib7oTyOuliOYekP_nA][web-n1][inet[/10.165.0.21:9300]]{client=true, data=false},[logstash-flume-n1-99849-4018][KNknQCs0TPi6-VgiC4f-8A][flume-n1][inet[/10.165.0.41:9300]]{client=true, data=false},}, reason: zen-disco-master_failed ([Dorma][DoIcmge9QdqxiRnmBjtVdg][build][inet[/10.165.1.140:9300]])
log4j, [2014-11-16T09:08:38.706]  INFO: org.elasticsearch.cluster.service: [logstash-grading-n1-76848-4038] detected_master [Masque][5mLN45_iTfq_YuFKV20OLg][build][inet[/10.165.1.140:9300]], added {[logstash-rest-n1-168118-4018][o6Gxxd1SQC2XchlsySaz8Q][rest-n1][inet[/10.165.0.31:9300]]{client=true, data=false},[logstash-web-n1-221006-4016][vRY0Ib7oTyOuliOYekP_nA][web-n1][inet[/10.165.0.21:9300]]{client=true, data=false},[Masque][5mLN45_iTfq_YuFKV20OLg][build][inet[/10.165.1.140:9300]],}, reason: zen-disco-receive(from master [[Masque][5mLN45_iTfq_YuFKV20OLg][build][inet[/10.165.1.140:9300]]])
log4j, [2014-11-16T09:08:38.719]  INFO: org.elasticsearch.cluster.service: [logstash-grading-n1-76848-4038] added {[logstash-flume-n1-99849-4018][KNknQCs0TPi6-VgiC4f-8A][flume-n1][inet[/10.165.0.41:9300]]{client=true, data=false},}, reason: zen-disco-receive(from master [[Masque][5mLN45_iTfq_YuFKV20OLg][build][inet[/10.165.1.140:9300]]])

Elastic search log:

[2014-11-16 09:08:36,248][INFO ][gateway                  ] [Masque] recovered [8] indices into cluster_state
...
[2014-11-16 09:09:39,085][INFO ][cluster.service          ] [Masque] added {[logstash-grading-n1-76848-4038][8szJ9egnQsaAvet6S10Tmw][grading-n1][inet[/10.165.0.71:9300]]{client=true, data=false},}, reason: zen-disco-receive(join from node[[logstash-grading-n1-76848-4038][8szJ9egnQsaAvet6S10Tmw][grading-n1][inet[/10.165.0.71:9300]]{client=true, data=false}])

6 - Restart logstash - events start working again

So the issue is I would like to not have to restart all my logstash instances in the cluster just because of an elastic search restart. Does anyone have any ideas whats going on? Am I missing some configuration?

解决方案

This looks like a known logstash issue:

"Infinite hang when Elasticsearch restarts"

https://github.com/elasticsearch/logstash/issues/1655

No known work around other than restarting logstash. I'd comment about your problem in this issue - the more people talking about the problem the faster it will get resolved.

这篇关于弹性搜索重新启动后，Logstash停止写入弹性搜索？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！