仅查看最后 5 分钟数据的 Kafka 配置 [英] Kafka Configuration for only seeing last 5 minutes of data

问题描述

抱歉，我是 Kafka 的新手，这个问题可能很简单，但我需要一些帮助.我没有弄清楚一些配置.有一个流数据，我希望消费者只看到生产者发送的最后 5 分钟的消息.我将 Confluent.Kafka 用于 .Net，

Sorry i am new in Kafka and this question migth be so easy but i need some help. i did not figure out some configurations. There is a stream data, i want Consumers to see only last 5 minutes of messages that procuders sent. I am using Confluent.Kafka for .Net,

var config = new Dictionary<string, object>{
                {"group.id","Test1Costumers"},
                {"bootstrap.servers",brokerEndpoint},
                { "auto.commit.interval.ms", 60000},
                { "auto.offset.reset", "earliest" }
            };

这是github示例中消费者的配置字典，另一个问题是我不想在主题中存储超过 5 分钟的消息，因为如果它们超过 5 分钟，我将不需要这些记录.

Here is config dictionary of Consumers in github example, another issue is i dont want to store messages in a topic more than 5 minutes cos i wont need those records if they are older than 5 minutes.

当我配置 server.properties;

When i configure server.properties;

# The minimum age of a log file to be eligible for deletion due to age
log.retention.ms=60000

一分钟后，文件当前使用的抛出错误

after a minute its throw error that file is currently uses

感谢您的帮助.

推荐答案

在 Kafka server.properties 中有一个名为 log.segment.bytes 的设置，它被设置为默认为 1GB.一旦日志段达到 1GB，它就会被关闭，只有在此之后保留才开始.例如.如果您每天生成 100MB 的消息，并且您的保留时间为 1 周，那么您实际上会在数据被删除之前保留大约 17 天.那是因为日志段需要 10 天才能填满 (1GB) 并且从那时起保留将开始.在您的情况下，我假设您没有更改 log.segment.bytes，但你的留存率很低.因此，由于日志段尚未关闭，它将无法清理数据.

In Kafka server.properties there's a setting called log.segment.bytes, which is set to 1GB by default. Once a log segment has reached 1GB, it is closed, and only after that the retention kicks in. E.g. if you are producing 100MB of message per day, and your retention is 1 week, you'd actually retain the data for around 17 days before it gets deleted. That's because the log segment will take 10 days to be full (1GB) and from that time retention will kick in. In your case, I'm assuming you haven't changed the value for log.segment.bytes, but your retention is very low. So, it won't be able to clean up the data as the log segment is not yet closed.

这篇关于仅查看最后 5 分钟数据的 Kafka 配置的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！