如何使用Kafka将日志保留在logstash中更长的时间? [英] How can I use Kafka to retain logs in logstash for longer period?
问题描述
当前,我使用redis-> s3->弹性搜索-> kibana堆栈来管道显示我的日志.但是由于弹性搜索中的大量数据,我最多可以保留7天的日志.
Currently I use redis -> s3 -> elastic search -> kibana stack to pipe and visualise my logs. But due to large volume of data in elastic search I can retain logs upto 7 days.
我想将kafka群集放入此堆栈中,并将日志保留更多天.我正在考虑跟随堆栈.
I want to bring kafka cluster in this stack and retain logs for more number of days. I am thinking of following stack.
app节点将日志传递到kafka-> kafka集群->弹性搜索集群-> kibana
app nodes piping logs to kafka -> kafka cluster -> elastics search cluster -> kibana
如何使用kafka将日志保留更多天?
How can I use kafka to retain logs for more number of days?
推荐答案
浏览Apache Kafka 经纪人配置,其中有两个属性可确定何时删除日志.一个按时间,另一个按空间.
Looking through the Apache Kafka broker configs, there are two properties that determine when a log will get deleted. One by time and the other by space.
log.retention.{ms,minutes,hours}
log.retention.bytes
还请注意,如果同时设置了log.retention.hours和log.retention.bytes字节,则当超出其中一个限制时,我们将删除一个段.
Also note that if both log.retention.hours and log.retention.bytes are both set we delete a segment when either limit is exceeded.
这两个指示何时在Kafka中删除日志. log.retention.bytes默认为-1,我很确定将其保留为-1仅允许时间配置单独确定何时删除日志.
Those two dictate when logs are deleted in Kafka. The log.retention.bytes defaults to -1, and I'm pretty sure leaving it to -1 allows only the time config to solely determine when a log gets deleted.
因此,要直接回答您的问题,请将log.retention.hours设置为您希望保留数据的小时数,并且不要更改log.retention.bytes配置.
So to directly answer your question, set log.retention.hours to however many hours you wish to retain your data and don't change the log.retention.bytes configuration.
这篇关于如何使用Kafka将日志保留在logstash中更长的时间?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!