Logstash和ElasticSearch之间丢失的文档 [英] Documents lost between Logstash and ElasticSearch
问题描述
Logstash正在侦听TCP 9140，其输入通过NxLog代理接收大约30个Windows Server 2008和30个Windows Server 2003事件（evts），并将其输出到弹性搜索。
这已经运行好几个星期了。我可以看到ElasticSearch创建每一天的索引，并且可以浏览文档，创建图表，一切都很好。
一个周末后，我意识到在星期五晚上9点的时候新事件停了。 没有网络问题，所有的服务器都正在运送他们的日志
我可以看到有很多与Watcher（弹性插件）相关的错误/警告没有许可证，但没有任何有趣的。我甚至可以看到与新索引相关的日志被自动创建。
所以我删除了插件，重新启动弹性和一切都好。
我不认为这是实际的问题，我认为弹性搜索被挂断了。
我有两个问题： / p>
1）我应该如何梳理这些条件。 （所有服务，没有文件索引）？
2）如果Logstash已启动并接受输入，但Elastic已关闭，那些从我的Windows服务器发送的事件会发生什么？从Nxlog的角度来看，这些日志正确地发送到logstash，没有理由重试，这些日志会丢失，永远？
谢谢！
Rodrigo。
如果ES因为任何原因（太忙GC等）而关机或挂断， ，那么logstash将重试几次，然后让它走，这意味着你会失去这些事件。
有很多方法可以减轻这个，但一个好的做法是持久存储事件（在DB或消息队列系统内），只有当它们成功发送到ES时才弹出。这种消息系统包括 Redis （使用列表或频道作为队列）， Apache Kafka （分布式日志）， Rabbitmq （分布式消息队列）等等
有很多方法可以配置这些技术和logstash，一个例子是这一个，它显示了与Kafka一起使用的Logstash。在这种情况下，这意味着Nxlog 将其日志发送到kafka 而不是直接到Logstash。然后Logstash 从Kafka主题中消耗日志。
您的里程将有所不同，但这里的主要思想是，如果您的日志不能发送到Elasticsearch，您的日志将不会丢失。这将回答您的第二个问题。
至于您的第一个，我建议安装其他ES插件，如 bigdesk 和总部和/或正式的 Marvel插件，这些都深入了解了什么目前正在Elasticsearch内进行。您可以快速检测出问题的位置，并采取行动。
I'm running and basic ELK stack. All three components runnning in the vm. Logstash is listening on TCP 9140 for its input receiving from about 30 Windows Server 2008s and 30 Windows Server 2003 Events (evts) via NxLog agents and its outputting to elasticsearch.
This has been running for a couple of weeks perfectly. I could see ElasticSearch creating indices for each day and could browse documents, create graphs , all good.
After a weekend I realized that at some point of Friday at 9pm, all new events stopped. No network issues, all servers were shipping their logs All I could see was A LOT of error/warnings related to Watcher (elastic plugin) being out of license but nothing interesting. I even could see logs related to new indices for the new day being automatically created.
So I removed the plugin, restarted elastic and all good. I don't think that was the actual problem, I think elasticsearch was hung up.
I have two questions:
1) How should I trobleshoot these conditions. (all services up, no documents being indexed)?
2) If Logstash is Up and accepting inputs but Elastic is down , what happens to those shipped events from my windows servers? Since the Nxlog point of view, those logs were correctly sent to logstash there would be no reason to retry and those logs would be lost , "forever" ?
thanks! Rodrigo.
If ES is down or hung up for whatever reason (too busy GC-ing, etc), then logstash will retry a few times and then let it go, which means you'll lose those events.
There are many ways to alleviate this but a good practice is to durably store the events (either inside a DB or a message queueing system) and pop them out only when they have been successfully sent to ES. Such messaging systems include Redis (using lists or channels as queues), Apache Kafka (distributed log), Rabbitmq (distributed message queues), etc
There are plenty of ways to configure these technologies together with logstash, one example would be this one which shows Logstash being used with Kafka. In your case, this would mean that Nxlog ships its logs to kafka instead of directly to Logstash. Then Logstash consumes logs from a Kafka topic.
Your mileage will vary, but the main idea here is that your logs will not be lost in case they cannot be sent to Elasticsearch. This would answer your second question.
As for your first one, I would advise to install other ES plugins, like bigdesk and HQ and/or the official Marvel plugin which all provide deep insights into what is currently going on inside Elasticsearch. You'll be able to quickly detect where the issue/s is/are and take action.
这篇关于Logstash和ElasticSearch之间丢失的文档的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
