卡夫卡只有一次消费保证 [英] Kafka only once consumption guarantee

问题描述

我在有关堆栈溢出的一些答案中发现，通常在网络上，人们都认为Kafka不支持消耗量确认，或者很难实现一次消耗量.

I see in some answers around stack-overflow and in general in the web the idea that Kafka does not support consumption acknowledge or that exactly once consumption is hard to achieve.

在以下条目中作为示例是否有理由在Kafka上使用RabbitMQ?，我可以阅读以下语句:

In the following entry as a sample Is there any reason to use RabbitMQ over Kafka?, I can read the following statements:

RabbitMQ将保留有关已使用/已确认/未确认消息的所有状态，而Kafka不会

RabbitMQ will keep all states about consumed/acknowledged/unacknowledged messages while Kafka doesn't

或

使用Kafka很难获得保证.

Exactly once guarantees are hard to get with Kafka.

这不是我通过阅读Kafka官方文档所了解的内容: https://kafka.apache.org/documentation/#design_consumerposition

This is not what I understand by reading the official Kafka documentation at: https://kafka.apache.org/documentation/#design_consumerposition

以前的文档指出，Kafka没有使用传统的确认实现(如RabbitMQ).相反，他们依赖于partition-consumer和offset的关系...

The previous documentation states that Kafka does not use a traditional acknowledge implementation (as RabbitMQ). Instead they rely on the relationship partition-consumer and offset...

这使得相当于消息确认的价格非常便宜

This makes the equivalent of message acknowledgements very cheap

有人可以解释为什么卡夫卡的仅一次消费保证"很难实现吗?与Kafka与其他更传统的Message Broker(如RabbitMQ)相比有何不同?我想念什么?

Could somebody please explain why "only once consumption guarantee" in Kafka is difficult to achieve? and How this differs from Kafka vs other more traditional Message Broker as RabbitMQ? What am I missing?

推荐答案

如果您的意思恰恰是这样的问题.您可能知道，Kafka使用者使用轮询机制，即使用者向服务器询问消息.同样，您需要回想一下消费者提交消息的偏移量，即它告诉集群下一个预期的偏移量是什么.所以，想象会发生什么.

If you mean exactly once the problem is like this. Kafka consumer as you may know use a polling mechanism, that is consumers ask the server for messages. Also, you need to recall that the consumer commit message offsets, that is, it tells the cluster what is the next expected offset. So, imagine what could happen.

对消息进行消费者轮询，并获取偏移量= 1的消息.

Consumer poll for messages and get message with offset = 1.

A)如果使用者在处理消息之前立即提交了偏移量，则它可能崩溃并且永远不会再收到该消息，因为它已被提交，在下一次轮询中，Kafka将返回偏移量= 2的消息.这就是他们所说的最多一次语义.

A) If consumer commit that offset immediately before processing the message, then it can crash and will never receive that message again because it was already committed, on next poll Kafka will return message with offset = 2. This is what they call at most once semantic.

B)如果使用者先处理消息然后提交偏移量，则可能发生的情况是，在处理了消息之后但在提交之前，使用者崩溃了，因此在这种情况下，下一次轮询将再次获得偏移量= 1的相同消息.该消息将被处理两次.这就是他们所谓的至少一次.

B) If consumer process the message first and then commit the offset, what could happen is that after processing the message but before committing, the consumer crashes, so in that case next poll will get again the same message with offset = 1 and that message will be processed twice. This is what they call at least once.

为了只实现一次，您需要处理消息并在原子操作中提交该偏移量，在该操作中您始终不执行任何操作.这不是那么容易.一种方法(如果可能)是将处理结果与生成该结果的消息的偏移量一起存储.然后，当消费者启动时，它会在Kafka之外查找最后处理的偏移量，并寻求该偏移量.

In order to achieve exactly once, you need to process the message and commit that offset in an atomic operation, where you always do both or none of them. This is not so easy. One way to do this (if possible) is to store the result of the processing along with the offset of the message that generated that result. Then, when consumer starts it looks for the last processed offset outside Kafka and seek to that offset.

这篇关于卡夫卡只有一次消费保证的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！