Kafka配置仅显示最后5分钟的数据 [英] Kafka Configuration for only seeing last 5 minutes of data

问题描述

对不起，我是Kafka的新人，这个问题很简单，但我需要一些帮助。
我没有弄清楚一些配置。
有一个流数据，
我希望消费者只能看到程序员发送的最后5分钟的消息。我正在使用.Net的Confluent.Kafka。

Sorry i am new in Kafka and this question migth be so easy but i need some help. i did not figure out some configurations. There is a stream data, i want Consumers to see only last 5 minutes of messages that procuders sent. I am using Confluent.Kafka for .Net,

var config = new Dictionary<string, object>{
                {"group.id","Test1Costumers"},
                {"bootstrap.servers",brokerEndpoint},
                { "auto.commit.interval.ms", 60000},
                { "auto.offset.reset", "earliest" }
            };

这是github示例中的Consumers的配置字典，
的另一个问题是我不想将消息存储在主题中的时间超过5分钟，因为如果它们的时间超过5分钟，我将不需要这些记录。

Here is config dictionary of Consumers in github example, another issue is i dont want to store messages in a topic more than 5 minutes cos i wont need those records if they are older than 5 minutes.

当我配置server.properties时；

When i configure server.properties;

# The minimum age of a log file to be eligible for deletion due to age
log.retention.ms=60000

一分钟后，当前正在使用文件的抛出错误

after a minute its throw error that file is currently uses

感谢您的帮助。

推荐答案

在Kafka server.properties 中有一个名为<$ c的设置$ c> log.segment.bytes ，默认设置为1GB。日志段达到1GB后，它将关闭，并且只有在保留期开始后才关闭。如果您每天产生100MB的邮件，并且保留时间为1周，则实际上该数据将保留大约17天，然后才能被删除。这是因为日志段将需要10天才能装满（1GB），并且从那时起保留将开始生效。就您而言，我假设您尚未更改 log.segment的值。个字节，但您的保留率很低。因此，由于日志段尚未关闭，因此无法清理数据。

In Kafka server.properties there's a setting called log.segment.bytes, which is set to 1GB by default. Once a log segment has reached 1GB, it is closed, and only after that the retention kicks in. E.g. if you are producing 100MB of message per day, and your retention is 1 week, you'd actually retain the data for around 17 days before it gets deleted. That's because the log segment will take 10 days to be full (1GB) and from that time retention will kick in. In your case, I'm assuming you haven't changed the value for log.segment.bytes, but your retention is very low. So, it won't be able to clean up the data as the log segment is not yet closed.

这篇关于Kafka配置仅显示最后5分钟的数据的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！