在KAFKA中使用后删除消息 [英] Delete message after consuming it in KAFKA

问题描述

我正在使用apache kafka制作和使用5GB大小的文件.我想知道是否有一种方法可以在使用完主题消息后自动将其删除.我有什么办法来跟踪已消耗的消息吗?我不想手动删除它.

I am using apache kafka to produce and consume a file 5GB in size. I want to know if there is a way where the message from the topic is automatically removed after it is consumed. Do I have any way to keep track of consumed messages? I don't want to delete it manually.

推荐答案

在Kafka中，所消费物品的责任是消费者的责任，这也是Kafka具有如此出色的水平可伸缩性的主要原因之一.

In Kafka, the responsibility of what has been consumed is the responsibility of the consumer and this is also one of the main reasons why Kafka has such great horizontal scalability.

使用高级使用者API将通过在Zookeeper中提交消耗的偏移量来自动为您执行此操作(或者特殊的Kafka主题正在使用更新的配置选项来跟踪消耗的消息).

Using the high level consumer API will automatically do this for you by committing consumed offsets in Zookeeper (or a more recent configuration option is using by a special Kafka topic to keep track of consumed messages).

简单的使用者API可让您处理如何以及在何处跟踪消耗的消息.

The simple consumer API make you deal with how and where to keep track of consumed messages yourself.

在Kafka中清除邮件是通过为主题指定保留时间或为其定义磁盘配额来自动完成的，因此对于您的一个5GB文件，在您定义的保留期限过去之后，该文件将被删除，无论它是否已消耗.

Purging of messages in Kafka is done automatically by either specifying a retention time for a topic or by defining a disk quota for it so for your case of one 5GB file, this file will be deleted after the retention period you define has passed, regardless of if it has been consumed or not.

这篇关于在KAFKA中使用后删除消息的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！