如何使用 Kafka 在 logstash 中保留日志更长时间? [英] How can I use Kafka to retain logs in logstash for longer period?

问题描述

目前我使用 redis -> s3 -> elastic search -> kibana stack 来管理和可视化我的日志.但是由于弹性搜索中的数据量很大，我可以将日志保留长达 7 天.

我想在这个堆栈中引入 kafka 集群并将日志保留更多天数.我正在考虑关注堆栈.

应用程序节点将日志传送到 kafka -> kafka 集群 -> elastics 搜索集群 -> kibana

如何使用 kafka 将日志保留更多天数?

解决方案

浏览 Apache Kafka brokerconfigs，有两个属性决定日志何时被删除.一个是时间，另一个是空间.

log.retention.{ms,minutes,hours}日志.retention.bytes

<块引用>

另请注意，如果同时设置了 log.retention.hours 和 log.retention.bytes，我们会在超过任一限制时删除一个段.

这两个决定了何时在 Kafka 中删除日志.log.retention.bytes 默认为 -1，我很确定将其保留为 -1 只允许时间配置单独确定日志何时被删除.

因此，要直接回答您的问题，请将 log.retention.hours 设置为您希望保留数据的小时数，并且不要更改 log.retention.bytes 配置.

Currently I use redis -> s3 -> elastic search -> kibana stack to pipe and visualise my logs. But due to large volume of data in elastic search I can retain logs upto 7 days.

I want to bring kafka cluster in this stack and retain logs for more number of days. I am thinking of following stack.

app nodes piping logs to kafka -> kafka cluster -> elastics search cluster -> kibana

How can I use kafka to retain logs for more number of days?

解决方案

Looking through the Apache Kafka broker configs, there are two properties that determine when a log will get deleted. One by time and the other by space.

log.retention.{ms,minutes,hours}
log.retention.bytes

Also note that if both log.retention.hours and log.retention.bytes are both set we delete a segment when either limit is exceeded.

Those two dictate when logs are deleted in Kafka. The log.retention.bytes defaults to -1, and I'm pretty sure leaving it to -1 allows only the time config to solely determine when a log gets deleted.

So to directly answer your question, set log.retention.hours to however many hours you wish to retain your data and don't change the log.retention.bytes configuration.

这篇关于如何使用 Kafka 在 logstash 中保留日志更长时间?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！