Kafka邮件重复 [英] Kafka having duplicate messages

问题描述

在生产或使用数据时我看不到任何故障，但是生产中有一堆重复的消息.对于一个大约有10万条消息的小话题，大约有4000条重复项，尽管就像我说的那样，没有失败，而且没有实现重试逻辑或未设置配置值.

I don't see any failure while producing or consuming the data however there are bunch of duplicate messages in production. For a small topic which gets around 100k messages, there are ~4k duplicates though like I said there is no failure and on top of that there is no retry logic implemented or config value is set.

我还检查那些重复消息的偏移值，每个偏移值都有不同的值，告诉我问题出在生产者中.

I also check offset values for those duplicate messages and each has distinct values which tells me that the issue is in producer.

任何帮助将不胜感激

推荐答案

了解有关在Kafka中传递消息的更多信息:

Read more about message delivery in kafka:

https://kafka.apache.org/08/design.html#semantics

因此，默认情况下，Kafka保证至少一次交付， 允许用户通过禁用最多实现一次投放 重试生产者并在处理之前提交其偏移量 一批消息.一次交货就需要与 目标存储系统，但Kafka提供了偏移量 可以直接实现.

So effectively Kafka guarantees at-least-once delivery by default and allows the user to implement at most once delivery by disabling retries on the producer and committing its offset prior to processing a batch of messages. Exactly-once delivery requires co-operation with the destination storage system but Kafka provides the offset which makes implementing this straight-forward.

可能您正在寻找恰好一次交货"(如jms)

Probably you are looking for "exactly once delivery" like in jms

https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-如何准确地从Kafka发消息?

有两种方法可以在数据期间准确获取一次语义 生产:1.每个分区使用一个单一写入器，每次得到一个 网络错误，请检查该分区中的最后一条消息，看看您的 上次写入成功2.在主目录中包含主键(UUID等). 消息并在消费者上进行重复数据删除.

There are two approaches to getting exactly once semantics during data production: 1. Use a single-writer per partition and every time you get a network error check the last message in that partition to see if your last write succeeded 2. Include a primary key (UUID or something) in the message and deduplicate on the consumer.

我们在系统中实现了第二点.

We implemented second point in our systems.

这篇关于Kafka邮件重复的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！