卡夫卡单一消费者失败 [英] Kafka single consumer failure in a group

问题描述

我正在探索0.8.1.1版的Kafka.

I am in the initial phases of exploring Kafka, version 0.8.1.1.

我已经成功运行了Consumer Group Example，它具有多个分区，并且在消费者之间很好地分配了消息.

I've successfully run the Consumer Group Example, with multiple partitions and its distributing messages among the consumers quite well.

我要运行的一个测试用例是组中的某个消费者突然死亡(例如，杀死-9).当我这样做时，我期望会发生重新平衡，但不会重新平衡.那么，我可以做这些事情之一吗?

One test case I wanted to run is when a consumer in the group dies suddenly (example, kill -9 ). When I do so, I expected rebalancing to occur, but its not happening. So, can I do one of these things?

1. 使用API​​触发重新平衡
2. 将kafka配置为等待一段时间以进行消费者活动，并假设不正常关闭了该活动，则会自动重新平衡.

这里的问题是，分配给失效的使用者的分区中的所有消息都保留在队列中，并且直到进行重新平衡时才进行处理.

The problem here is, all the messages in the partitions assigned to the dead Consumer remains in the queue and is never processed until rebalancing occurs.

推荐答案

重新平衡将自动发生，可以在使用者配置(zookeeper.session.timeout.ms)中进行设置.根据文档

The Rebalance will happen automatically which can be set in the consumer config ( zookeeper.session.timeout.ms ). As per the documentaion

zookeeper.session.timeout.ms :ZooKeeper会话超时.如果消费者在这段时间内没有对动物园管理员心跳，则认为它已经死了，并且将发生重新平衡.默认值为6000毫秒

zookeeper.session.timeout.ms : ZooKeeper session timeout. If the consumer fails to heartbeat to zookeeper for this period of time it is considered dead and a rebalance will occur. default value is 6000 ms

在超时间隔之后，同一组中的其他实时使用者将开始接收消息.

The other live consumer in the same group will start to recieve the message after the timeout interval.

根据您的要求配置此超时值.

Configure this timeout value as per your requirements.

还有来自kafka文档的更多信息:

Also some more info from the kafka documentation:

消费者重新平衡失败(您将看到ConsumerRebalanceFailedException):这是由于当两个消费者试图拥有相同的主题分区时发生冲突.日志将显示导致冲突的原因(搜索"conflict in").

Consumer rebalancing fails (you will see ConsumerRebalanceFailedException): This is due to conflicts when two consumers are trying to own the same topic partition. The log will show you what caused the conflict (search for "conflict in ").

1. 如果您的消费者订阅了许多主题，而您的ZK服务器繁忙，则可能是由于消费者没有足够的时间来查看同一组中所有消费者的一致视图所致.在这种情况下，请尝试增加rebalance.max.retries和rebalance.backoff.ms.
2. 另一个原因可能是其中一个消费者被杀害.在重新平衡过程中，其他消费者不会意识到消费者在zookeeper.session.timeout.ms时间之后就走了.在这种情况下，请确保rebalance.max.retries * rebalance.backoff.ms>zookeeper.session.timeout.ms.

这篇关于卡夫卡单一消费者失败的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！