Kafka - 经纪人:组协调员不可用 [英] Kafka - Broker: Group coordinator not available

问题描述

我有以下结构:

zookeeper: 3.4.12
kafka: kafka_2.11-1.1.0
server1: zookeeper + kafka
server2: zookeeper + kafka
server3: zookeeper + kafka

通过 kafka-topics shell 脚本创建具有复制因子 3 和分区 3 的主题.

Created topic with replication factor 3 and partitions 3 by kafka-topics shell script.

./kafka-topics.sh --create --zookeeper localhost:2181 --topic test-flow --partitions 3 --replication-factor 3

并使用组 localConsumers.当leader没问题时它工作正常.

And use group localConsumers. it works fine when leader is ok.

./kafka-topics.sh --describe --zookeeper localhost:2181 --topic test-flow
Topic:test-flow PartitionCount:3    ReplicationFactor:3 Configs:
    Topic: test-flow    Partition: 0    Leader: 3   Replicas: 3,2,1 Isr: 3,2,1
    Topic: test-flow    Partition: 1    Leader: 1   Replicas: 1,3,2 Isr: 1,3,2
    Topic: test-flow    Partition: 2    Leader: 2   Replicas: 2,1,3 Isr: 2,1,3

消费者日志

Received FindCoordinator response ClientResponse(receivedTimeMs=1529508772673, latencyMs=217, disconnected=false, requestHeader=RequestHeader(apiKey=FIND_COORDINATOR, apiVersion=1, clientId=consumer-1, correlationId=0), responseBody=FindCoordinatorResponse(throttleTimeMs=0, errorMessage='null', error=NONE, node=myserver3:9092 (id: 3 rack: null)))

但是如果领导者宕机了 - 我在消费者中得到错误(systemctl stop kafka):

But if leader is down - I get the error in consumer (systemctl stop kafka):

节点 3 不可用.好的

Node 3 is unavailable. ok

./kafka-topics.sh --describe --zookeeper localhost:2181 --topic test-flow
Topic:test-flow PartitionCount:3    ReplicationFactor:3 Configs:
    Topic: test-flow    Partition: 0    Leader: 2   Replicas: 3,2,1 Isr: 2,1
    Topic: test-flow    Partition: 1    Leader: 1   Replicas: 1,3,2 Isr: 1,2
    Topic: test-flow    Partition: 2    Leader: 2   Replicas: 2,1,3 Isr: 2,1

消费者日志

Received FindCoordinator response 
ClientResponse(receivedTimeMs=1529507314193, latencyMs=36, 
disconnected=false, 
requestHeader=RequestHeader(apiKey=FIND_COORDINATOR, apiVersion=1, 
clientId=consumer-1, correlationId=149), 
responseBody=FindCoordinatorResponse(throttleTimeMs=0, 
errorMessage='null', error=COORDINATOR_NOT_AVAILABLE, node=:-1 (id: -1 
rack: null)))

- Group coordinator lookup failed: The coordinator is not available.
- Coordinator discovery failed, refreshing metadata

消费者无法连接，直到领导者关闭或与另一个消费者组重新连接.

Consumer unable to connect until leader is down or reconnect with another consumer group.

不明白为什么会这样?消费者应该重新平衡到另一个经纪人，但事实并非如此.

Can't understand why it happens? Consumer should be rebalanced to another broker, but it doesn't.

推荐答案

尝试在 server.conf 中添加属性并清理 zookeeper 缓存.应该有帮助

Try to add properties into the server.conf and clean zookeeper cache. It should help

offsets.topic.replication.factor=3
default.replication.factor=3

此问题的根本原因是无法在节点之间分配主题偏移量.

Root cause of this issue is impossibility to distribute topic offsets between nodes.

自动生成主题:__consumer_offsets

Auto generated topic: __consumer_offsets

您可以通过以下方式查看

You can check it by

$ ./kafka-topics.sh --describe --zookeeper localhost:2181 --topic __consumer_offsets

关注这篇文章:https://kafka.apache.org/documentation/#prodconfig

默认情况下，它使用 RF - 1 创建 __consumer_offsets

by default it creates __consumer_offsets with RF - 1

重要的是在 kafka/cluster 启动之前配置复制因子.否则，它可能会在重新配置实例时带来一些问题，例如您的情况.

Important thing is to configure replication factor before the kafka/cluster start. Otherwise it can bring some issues with re configuring instances like in your case.

这篇关于Kafka - 经纪人:组协调员不可用的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！