Kafka 弹性 - 组协调员 [英] Kafka Resiliency - Group Coordinator

问题描述

据我所知，其中一位经纪人被选为负责消费者再平衡的组协调员.

As I understand, one of the brokers is selected as the group coordinator which takes care of consumer rebalancing.

Discovered coordinator host:9092 (id: 2147483646 rack: null) for group good_group

我有 3 个节点，复制因子为 3 和 3 个分区.一切都很好，当我在非协调器节点上杀死 kafka 时，消费者仍在接收消息.

I have 3 nodes with replication factor of 3 and 3 partitions. Everything is great and when I kill kafka on non-coordinator nodes, consumer is still receiving messages.

但是当我使用协调器杀死该特定节点时，不会发生重新平衡，并且我的 Java 消费者应用程序未收到任何消息.

But when I kill that specific node with coordinator, rebalancing is not happening and my java consumer app does not receive any messages.

2018-05-29 16:34:22.668 INFO  AbstractCoordinator:555 - Discovered coordinator host:9092 (id: 2147483646 rack: null) for group good_group.
2018-05-29 16:34:22.689 INFO  AbstractCoordinator:600 - Marking the coordinator host:9092 (id: 2147483646 rack: null) dead for group good_group
2018-05-29 16:34:22.801 INFO  AbstractCoordinator:555 - Discovered coordinator host:9092 (id: 2147483646 rack: null) for group good_group.
2018-05-29 16:34:22.832 INFO  AbstractCoordinator:600 - Marking the coordinator host:9092 (id: 2147483646 rack: null) dead for group good_group
2018-05-29 16:34:22.933 INFO  AbstractCoordinator:555 - Discovered coordinator host:9092 (id: 2147483646 rack: null) for group good_group.
2018-05-29 16:34:23.044 WARN  ConsumerCoordinator:535 - Auto offset commit failed for group good_group: Offset commit failed with a retriable exception. You should retry committing offsets. 

我是不是做错了什么，有没有办法解决这个问题?

Am I doing something wrong and is there a way around this?

推荐答案

但是当我使用协调器杀死该特定节点时，不会发生重新平衡并且我的 Java 消费者应用程序未收到任何消息.

But when I kill that specific node with coordinator, rebalancing is not happening and my java consumer app does not receive any messages.

组协调器接收来自消费者组中所有消费者的心跳.它维护一个活跃消费者的列表，并在这个列表的变化上启动重新平衡.然后组长执行重新平衡活动.

The group coordinator receives heartbeats from all consumers in the consumer group. It maintains a list of active consumers and initiates the rebalancing on the change of this list. Then the group leader executes the rebalance activity.

这就是为什么如果您杀死组协调器，重新平衡将停止的原因.

That's why the rebalancing will stop if you kill the group coordinator.

更新

如果群协调器broker关闭，会通知Zookeeper，选举开始自动从活跃的broker中提拔新的群协调器.所以与组协调员无关.让我们看看日志:

In the case that the group coordinator broker shutdowns, the Zookeeper will be notified and the election starts to promote a new group coordinator from the active brokers automatically. So nothing to do with group coordinator. Let's see the log:

2018-05-29 16:34:23.044 WARN  ConsumerCoordinator:535 - Auto offset commit failed for group good_group: Offset commit failed with a retriable exception. You should retry committing offsets.

内部主题__consumer_offset的复制因子可能有默认值1.你能检查一下default.replication.factor和offsets.topic的值是多少.replication.factor 位于 server.properties 文件中.如果默认值为 1，则应将其更改为更大的值.如果不这样做，组协调器将关闭，导致偏移管理器在没有备份的情况下停止.因此无法完成提交偏移量的活动.

The replication factor of internal topic __consumer_offset probably has the default value 1. Can you check what value of default.replication.factor and offsets.topic.replication.factor are in the server.properties files. If the values is 1 by default, it should be changed to bigger one. Failing to do so, the group coordinator shutdowns causing offset manager stops without backup. So the activity of committing offsets can not be done.

这篇关于Kafka 弹性 - 组协调员的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！