REDIS 在 ELK 堆栈中的意义是什么? [英] What is the point of REDIS in ELK stack?

问题描述

我目前使用 filebeat 作为日志传送器的架构，它将日志发送到日志存储索引器实例，然后发送到 AWS 中的托管弹性搜索.由于持续的 TCP 连接，我无法使用 AWS ELB 多个日志存储索引器实例进行负载平衡，因为 filebeats 总是选择实例并将其发送到那里.所以我决定使用redis.现在看到在 ELK 堆栈中扩展 redis 并使其成为高可用组件有多么困难，我想问一下 redis 的意义何在.我读了一百万次它充当缓冲区，但是如果如果 logstash 无法处理负载，filebeats 停止向logstash发送日志，为什么我们甚至需要缓冲区.Filebeat 足够聪明，知道停止发送日志.Logstash 足够聪明，可以在弹性搜索出现故障时停止向弹性搜索发送日志.所以管道停止了.我真的不明白 redis 在每个标准 ELK 架构中都充当缓冲区.

I currently have architecture with filebeat as the log shipper, which sends logs to log stash indexer instance and then to managed elastic search in AWS. Due to persistent TCP connections, I cannot load balance using AWS ELB multiple log stash indexer instances since filebeats always picks on of the instances and sends it there. So I decided to use redis. Now seeing how difficult it is to scale redis and make it highly available compontent in ELK stack I want to ask what is even the point of redis. I read a million times it acts as a buffer, but if filebeats stops sending logs to logstash if logstash can't handle the load, why do we even need a buffer. Filebeat is smart enough to know to stop sending logs. Logstash is smart enough to stop sending logs to elastic search if elastic search goes down. So the pipeline stops. I really don't understand of the redis acting as a buffer in every standard ELK architecture.

推荐答案

Redis or Kafka or XYZ 可以作为 ELK 堆栈中的缓冲区，正如您所注意到的.

Redis or Kafka or XYZ can be used as buffer in the ELK stack as you've rightly noticed.

ES 人员发表了一篇博文 昨天关于在管道中使用 Kafka，但它也可能是 Redis 或 XYZ.他们很好地说明了何时可能需要和何时不需要这样的缓冲区.

The ES folks published a blog post yesterday about using Kafka in the pipeline, but it could as well have been Redis or XYZ. They make a good point about WHEN such a buffer could be needed and when it is not.

拥有这样一个缓冲区是个好主意

It is a good idea to have such a buffer in order to

1. 处理事件高峰
2. 处理可能无法访问的 ES 集群

如果您没有预料到此类行为，即您知道

If you don't anticipate such behaviors, i.e. you know

1. 您的活动将始终以相同的速度和/或
2. 如果您需要升级 ES 集群，您可以稍后再发送日志

...那你就不需要这样的缓冲区了.更重要的是，这将减少您需要管理、监控和维护的软件.

...then you don't need such a buffer. What's more, that will be one less piece of software you need to manage, monitor and maintain.

就 Elastic Stack 生态系统而言，没有一刀切的方法，它始终取决于您的具体用例和要求.您需要问问自己什么对您、您的系统和您的用户最重要，然后相应地设计您的解决方案.

When it comes to the Elastic Stack ecosystem, there's no one-size-fits-all approach, it always depends on your precise use case and requirements. You need to ask yourself what is important to you, your system(s) and your users and then design your solution accordingly.

这篇关于REDIS 在 ELK 堆栈中的意义是什么?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！