当其中一个经纪人关闭时，Kafka 消费者不会收到数据 [英] Kafka Consumer does not receive data when one of the brokers is down

问题描述

Kafka 快速入门
在 RHEL v6.9 上使用 Kafka v2.1.0

Kafka Quickstart
Using Kafka v2.1.0 on RHEL v6.9

当其中一个 Kafka 代理宕机时，消费者无法接收数据.

Consumer fails to receive data when one of the Kafka brokers is down.

执行的步骤:
1.启动zookeeper
2. 启动 Kafka-Server0 (localhost:9092, kafkalogs1)
3. 启动 Kafka-Server1 (localhost:9094, kafkalog2)
4.创建主题test1"，分区数=1，复制因子=2
5. 为主题test1"运行生产者
6. 运行消费者
7.从生产者发送消息
8.在消费者端接收消息.

Steps performed:
1. Start zookeeper
2. Start Kafka-Server0 (localhost:9092, kafkalogs1)
3. Start Kafka-Server1 (localhost:9094, kafkalog2)
4. Create topic "test1", num of partitions = 1, replication factor = 2
5. Run producer for topic "test1"
6. Run consumer
7. Send messages from the producer
8. Receive messages on the consumer side.

以上所有步骤都没有任何问题.

All the above steps worked without any issues.

当我关闭 Kafka-Server0 时，消费者停止从生产者获取数据.当我重新启动 Kafka-Server0 时，消费者开始从它停止的地方获取消息.

When I shutdown Kafka-Server0, the consumer stops getting data from Producer. When I bring back up Kafka-Server0, the consumer starts to get messages from where it left off.

这些是使用的命令

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test1
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test1
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 2 --partitions 1 --topic test1

当我使用 --bootstrap-server 选项中指定的两个服务器运行消费者时，行为是相同的(消费者端没有收到消息).

The behavior is the same (no message received on the consumer side) when I run the consumer with two servers specified in the --bootstrap-server option.

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092,localhost:9094 --topic test1

知道为什么即使主题 test1 的复制因子设置为 2，当 server0 关闭时消费者也停止接收消息吗?

Any idea why the consumer stops getting messages when server0 is down even though the replication factor for the topic test1 was set to 2?

已经有一个类似的问题，但没有完全回答Kafka 0.10 快速入门:消费者在主要"时失败经纪人被打倒

There is a similar question already but it was not answered completely Kafka 0.10 quickstart: consumer fails when "primary" broker is brought down

推荐答案

如果offsets主题不可用，则无法消费.

If the offsets topic is unavailable, you cannot consume.

查看 server.properties 文件，并查看上面的注释，并相应增加(仅适用于主题不存在的情况)

Look at the server.properties file for these, and see the comment above, and increase accordingly (only applies if topic doesn't already exist)

# The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state"
# For anything other than development testing, a value greater than 1 is recommended for to ensure availability such as 3.
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1

根据您之前的问题，看起来像只有一个副本

According to your previous question, looks like it only has one replica

了解如何增加现有主题的复制因子

这篇关于当其中一个经纪人关闭时，Kafka 消费者不会收到数据的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！