Kafka 0.10 快速入门:消费者在“主要"时失败经纪人被打倒 [英] Kafka 0.10 quickstart: consumer fails when "primary" broker is brought down

问题描述

所以我正在按照主要文档尝试 kafka 快速入门.按照说明获取了多集群示例的所有设置和测试，并且可以正常工作.例如，关闭一个代理，生产者和消费者仍然可以发送和接收.

So I'm trying the kafka quickstart as per the main documentation. Got the multi-cluster example all setup and test per the instructions and it works. For example, bringing down one broker and the producer and consumer can still send and receive.

但是，根据示例，我们设置了 3 个代理并关闭了代理 2(代理 ID = 1).现在，如果我再次调出所有经纪人，但我调低了经纪人 1(经纪人 id = 0)，消费者就会挂起.这只会发生在代理 1 (id = 0) 上，不会发生在代理 2 或 3.我正在 Windows 7 上测试.

However, as per the example, we setup 3 brokers and we bring down broker 2 (with broker id = 1). Now if I bring up all brokers again, but I bring down broker 1 (with broker id = 0), the consumer just hangs. This only happens with broker 1 (id = 0), does not happen with broker 2 or 3. I'm testing this on Windows 7.

经纪人 1 有什么特别之处吗?查看配置，除了 id、端口号和日志文件位置之外，所有 3 个代理之间的配置完全相同.

Is there something special here with broker 1? Looking at the config they are exactly the same between all 3 brokers except the id, port number and log file location.

我认为这只是提供的控制台消费者的问题，它不接受代理列表，所以我根据他们的文档使用默认设置编写了一个简单的 Java 消费者，但在bootstrap.xml"中指定了代理列表.服务器"属性，但没有骰子，仍然会遇到同样的问题.

I thought it is just a problem with the provided console consumer which doesn't take a broker list, so I wrote a simple java consumer as per their documentation using the default setup but specify the list of brokers in the "bootstrap.servers" property, but no dice, still get the same problem.

当我启动 broker 1 (broker id = 0) 时，消费者将继续工作.对于消费者来说，这不是高可用/容错行为......有关如何设置 HA/容错消费者的任何帮助?

The moment I startup broker 1 (broker id = 0), the consumers will just resume working. This isn't a highly available/fault tolerant behavior for the consumer... any help on how to setup a HA/fault tolerant consumer?

制作人似乎没有问题.

推荐答案

如果按照快速入门，创建的主题应该只有一个分区和一个副本，默认情况下托管在第一个代理中，即代理 1.这就是为什么当你关闭这个代理时消费者失败的原因.

If you follow the quick-start, the created topic should have only one partition with one replica which is hosted in the first broker by default, namely broker 1. That's why the consumer got failed when you brought down this broker.

尝试创建具有多个副本的主题(创建主题时指定--replication-factor)并重新运行测试以查看它是否带来更高的可用性.

Try to create a topic with multiple replicas(specifying --replication-factor when creating topic) and rerun your test to see whether it brings higher availability.

这篇关于Kafka 0.10 快速入门:消费者在“主要"时失败经纪人被打倒的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！