REDIS在ELK堆栈中的意义是什么？ [英] What is the point of REDIS in ELK stack?

问题描述

我目前拥有文件发送器作为日志传送器的体系结构，该结构将日志发送到日志存储索引器实例，然后发送到AWS中的托管弹性搜索。由于存在持续的TCP连接，因此我无法使用AWS ELB多个日志存储索引器实例进行负载平衡，因为文件信号始终会选择这些实例并将其发送到那里。所以我决定使用redis。现在，了解扩展redis并使之在ELK堆栈中具有高可用性的组件有多么困难，我想问一下redis的意义是什么。我读了一百万遍，它充当缓冲区，但是如果logbeat不能处理负载，如果文件信号停止将日志发送到logstash，为什么我们甚至需要缓冲区。 Filebeat足够聪明，足以知道它停止发送日志。如果弹性搜索失败，Logstash足够聪明，可以停止将日志发送到弹性搜索。因此，管道停止了。我真的不明白redis在每种标准ELK体系结构中都充当缓冲区。

I currently have architecture with filebeat as the log shipper, which sends logs to log stash indexer instance and then to managed elastic search in AWS. Due to persistent TCP connections, I cannot load balance using AWS ELB multiple log stash indexer instances since filebeats always picks on of the instances and sends it there. So I decided to use redis. Now seeing how difficult it is to scale redis and make it highly available compontent in ELK stack I want to ask what is even the point of redis. I read a million times it acts as a buffer, but if filebeats stops sending logs to logstash if logstash can't handle the load, why do we even need a buffer. Filebeat is smart enough to know to stop sending logs. Logstash is smart enough to stop sending logs to elastic search if elastic search goes down. So the pipeline stops. I really don't understand of the redis acting as a buffer in every standard ELK architecture.

推荐答案

Redis或Kafka或XYZ可以用作ELK中的缓冲区如您所知，

Redis or Kafka or XYZ can be used as buffer in the ELK stack as you've rightly noticed.

ES伙计们发表了博客帖子昨天是关于在管道中使用Kafka的信息，但也可能是Redis或XYZ。他们对何时何时需要这样的缓冲区以及何时不需要这样的缓冲区提出了很好的意见。

The ES folks published a blog post yesterday about using Kafka in the pipeline, but it could as well have been Redis or XYZ. They make a good point about WHEN such a buffer could be needed and when it is not.

最好有这样一个缓冲区

1. 处理事件尖峰


2. 处理可能无法访问的ES群集
>

1. handle event spikes
2. deal with a potentially unreachable ES cluster

如果您没有想到这种行为，即知道

If you don't anticipate such behaviors, i.e. you know

1. 您的事件将始终以相同的速度发生，并且/或者


2. 您还可以稍后再发送日志，以防万一您需要升级ES群集

...那么您就不需要这样的缓冲区了。而且，管理，监视和维护所需的软件也将减少。

...then you don't need such a buffer. What's more, that will be one less piece of software you need to manage, monitor and maintain.

在Elastic Stack生态系统中，没有任何一种规格可以万能的方法，它始终取决于您的确切用例和要求。您需要问问自己，什么对您，您的系统和用户很重要，然后相应地设计解决方案。

When it comes to the Elastic Stack ecosystem, there's no one-size-fits-all approach, it always depends on your precise use case and requirements. You need to ask yourself what is important to you, your system(s) and your users and then design your solution accordingly.

这篇关于REDIS在ELK堆栈中的意义是什么？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！