如果Kafka Consumer实例死亡，会发生什么情况? [英] What happens if a Kafka Consumer instance dies?

问题描述

• Kafka Broker具有3个分区.
• Kafka Consumer实例的计数为3.
• 突然，一个Consumer实例死亡.

我知道，如果一个Kafka Consumer实例死亡，则Kafka Broker正在重新平衡，并且另一个Consumer实例被分配到该分区.

I know that if a Kafka Consumer instance dies, the Kafka Broker is rebalancing and another consumer instance gets allocated to that partition.

我想知道另一个实例消耗了它最初消耗的所有分区，然后分配并消耗了死分区是否正确.

I wonder if it is correct to assume that another instance consumes all of the partition it originally consumes and then allocates and consumes dead partitions.

(我是否必须在客户端代码中实现ConsumerRebalanceListener?)

(And do I have to implement ConsumerRebalanceListener in client code?)

在这种情况下，使用该消息是否会有任何延迟?

If this is the case, can there be any delay in consuming the message?

谢谢.

推荐答案

如果我没记错的话，重新平衡会中断您对其他消费者的处理.

If I am not mistaken rebalancing will interrupt the processing of your different consumers.

如果您要在每个批次的末尾提交偏移量，则意味着将重新处理批次中所有已处理的数据.

If you are commiting your offset at the end of each batch, that means that all the data that had already been processed in your batch will be reprocessed again.

为避免这种情况，您可以使用

To avoid that you can either use consumer.commitAsync() that allows you to commit your offset in the middle of a batch processing, or implement ConsumerRebalanceListener as you infered.

public void onPartitionsRevoked(Collection<TopicPartition> partitions)

在重新平衡开始之前和使用者停止使用消息之后调用.这是您要提交偏移量的位置，因此下一个获得此分区的人将知道从何处开始.

Called before the rebalancing starts and after the consumer stopped consuming messages. This is where you want to commit offsets, so whoever gets this partition next will know where to start.

来自权威指南卡夫卡

回答最后一个问题:是的，重新平衡意味着延迟使用消息.

And to answer your last question : Yes, rebalancing implies a delay in consuming messages.

这篇关于如果Kafka Consumer实例死亡，会发生什么情况?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！