如果 Kafka Consumer 实例死亡会发生什么? [英] What happens if a Kafka Consumer instance dies?

问题描述

• Kafka Broker 有 3 个分区.
• Kafka 消费者实例的数量为 3.
• 突然，一个消费者实例死亡.

我知道如果 Kafka Consumer 实例死亡，Kafka Broker 正在重新平衡，另一个消费者实例会被分配到该分区.

I know that if a Kafka Consumer instance dies, the Kafka Broker is rebalancing and another consumer instance gets allocated to that partition.

我想知道假设另一个实例消耗了它最初消耗的所有分区然后分配和消耗死分区是否正确.

I wonder if it is correct to assume that another instance consumes all of the partition it originally consumes and then allocates and consumes dead partitions.

(我是否必须在客户端代码中实现 ConsumerRebalanceListener?)

(And do I have to implement ConsumerRebalanceListener in client code?)

如果是这种情况，消费消息会不会有延迟?

If this is the case, can there be any delay in consuming the message?

谢谢.

推荐答案

如果我没记错的话，重新平衡会中断不同消费者的处理.

If I am not mistaken rebalancing will interrupt the processing of your different consumers.

如果您在每个批次结束时提交偏移量，这意味着您批次中已经处理过的所有数据都将再次重新处理.

If you are commiting your offset at the end of each batch, that means that all the data that had already been processed in your batch will be reprocessed again.

为了避免这种情况，您可以使用 consumer.commitAsync() 允许您在批处理过程中提交偏移量，或实现 ConsumerRebalanceListener 正如你所推断的那样.

To avoid that you can either use consumer.commitAsync() that allows you to commit your offset in the middle of a batch processing, or implement ConsumerRebalanceListener as you infered.

public void onPartitionsRevoked(Collection partitions)

在重新平衡开始之前和消费者停止消费消息之后调用.这是您要提交偏移量的地方，因此接下来获得此分区的人将知道从哪里开始.

Called before the rebalancing starts and after the consumer stopped consuming messages. This is where you want to commit offsets, so whoever gets this partition next will know where to start.

来自 Kafka 权威指南

回答你的最后一个问题:是的，重新平衡意味着消费消息的延迟.

And to answer your last question : Yes, rebalancing implies a delay in consuming messages.

这篇关于如果 Kafka Consumer 实例死亡会发生什么?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！