当其中一个代理关闭时，Kafka Consumer无法接收数据 [英] Kafka Consumer does not receive data when one of the brokers is down

问题描述

Kafka快速入门
在RHEL v6.9上使用Kafka v2.1.0

Kafka Quickstart
Using Kafka v2.1.0 on RHEL v6.9

其中一个Kafka代理关闭时，消费者无法接收数据.

Consumer fails to receive data when one of the Kafka brokers is down.

执行的步骤:
1.启动Zookeeper
2.启动Kafka-Server0(localhost:9092，kafkalogs1)
3.启动Kafka-Server1(localhost:9094，kafkalog2)
4.创建主题"test1"，分区数= 1，复制因子= 2
5.为主题"test1"运行生产者
6.运行消费者
7.发送来自生产者的消息
8.在用户方接收消息.

Steps performed:
1. Start zookeeper
2. Start Kafka-Server0 (localhost:9092, kafkalogs1)
3. Start Kafka-Server1 (localhost:9094, kafkalog2)
4. Create topic "test1", num of partitions = 1, replication factor = 2
5. Run producer for topic "test1"
6. Run consumer
7. Send messages from the producer
8. Receive messages on the consumer side.

以上所有步骤均正常进行.

All the above steps worked without any issues.

当我关闭Kafka-Server0时，使用者停止从Producer获取数据.当我恢复Kafka-Server0时，使用者开始从中断的地方获取消息.

When I shutdown Kafka-Server0, the consumer stops getting data from Producer. When I bring back up Kafka-Server0, the consumer starts to get messages from where it left off.

这些是使用的命令

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test1
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test1
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 2 --partitions 1 --topic test1

当我使用--bootstrap-server选项中指定的两台服务器运行使用者时，行为是相同的(使用者端未收到任何消息).

The behavior is the same (no message received on the consumer side) when I run the consumer with two servers specified in the --bootstrap-server option.

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092,localhost:9094 --topic test1

有什么主意，即使主题test1的复制因子设置为2，为什么当server0关闭时使用者仍然停止获取消息?

Any idea why the consumer stops getting messages when server0 is down even though the replication factor for the topic test1 was set to 2?

已经有一个类似的问题，但是没有完全回答 Kafka 0.10快速入门:当主要"消费者选择失败时，消费者失败.经纪人被关闭

There is a similar question already but it was not answered completely Kafka 0.10 quickstart: consumer fails when "primary" broker is brought down

推荐答案

如果offsets主题不可用，则无法使用.

If the offsets topic is unavailable, you cannot consume.

查看这些文件的 server.properties 文件，并查看上面的注释，并相应地增加(仅在主题不存在的情况下适用)

Look at the server.properties file for these, and see the comment above, and increase accordingly (only applies if topic doesn't already exist)

# The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state"
# For anything other than development testing, a value greater than 1 is recommended for to ensure availability such as 3.
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1

根据您之前的问题，看起来像只有一个副本

According to your previous question, looks like it only has one replica

了解如何增加现有主题的复制因子

这篇关于当其中一个代理关闭时，Kafka Consumer无法接收数据的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！