测试“故障转移"功能.在卡夫卡 [英] Testing "fail-over" on Kafka

问题描述

操作系统:Windows 10

OS: Windows 10

3个 ZooKeeper 实例(已通过 v3.5.6 和 v.3.4.14 测试):
(1)apache-zookeeper-3.5.6-bin_1
(2)apache-zookeeper-3.5.6-bin_2(副本1)
(3)apache-zookeeper-3.5.6-bin_3(副本1)

3 ZooKeeper instances downloaded from Apache(tested with v3.5.6 and v.3.4.14):
(1) apache-zookeeper-3.5.6-bin_1
(2) apache-zookeeper-3.5.6-bin_2 (Copy of 1)
(3) apache-zookeeper-3.5.6-bin_3 (Copy of 1)

zoo.cfg:
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/tmp/zookeeper_3.4.14_1
clientPort=2181
admin.serverPort=10081
server.1=localhost:2881:3881
server.2=localhost:2882:3882
server.3=localhost:2883:3883
4lw.commands.whitelist=*

zoo.cfg:
...
dataDir=/tmp/zookeeper_3.4.14_2
clientPort=2182
admin.serverPort=10082
...

zoo.cfg:
...
dataDir=/tmp/zookeeper_3.4.14_3
clientPort=2183
admin.serverPort=10083
...

dataDir中的

myid文件，其值分别为1,2和3

myid file in dataDir with values 1,2 and 3 respectively

2个 Kafka 实例:
(1)kafka_2.12-2.3.0_1
(2)kafka_2.12-2.3.0_2(1个副本)

2 Kafka instances:
(1) kafka_2.12-2.3.0_1
(2) kafka_2.12-2.3.0_2 (Copy of 1)

server.properties:
...
broker.id=1
listeners=PLAINTEXT://:9091
log.dirs=/tmp/kafka-logs-1
zookeeper.connect=localhost:2181,localhost:2182,localhost:2183
...

server.properties:
...
broker.id=2
listeners=PLAINTEXT://:9092
log.dirs=/tmp/kafka-logs-2
zookeeper.connect=localhost:2181,localhost:2182,localhost:2183
...

春天

spring-boot-starter- * 2.2.0.RELEASE
spring-kafka-2.3.1.RELEASE

Spring

spring-boot-starter-* 2.2.0.RELEASE
spring-kafka-2.3.1.RELEASE

=====================================================================

=====================================================================

与设置1相同，唯一的区别在于，我使用的是 ZooKeeper ，而不是使用从 Apache 下载的 ZooKeeper Kafka 附带的.

Same as set-up 1, the only difference being that instead of using the ZooKeeper downloaded from Apache, i am using the ZooKeeper that comes with Kafka.

=====================================================================

=====================================================================

问题是当我放下1个 Kafka 时:

The issue is that when i bring 1 Kafka down:

=>设置1将不进行故障转移，这意味着当我生成一条消息时，启动的 Kafka 不会接收到该消息=>设置2将进行故障转移，这意味着当我生成一条消息时，启动的 Kafka 将会收到消息

=> Set-up 1 will not fail-over, meaning that when i produce a message, the Kafka that is up is not receiving the message => Set-up 2 will fail-over, meaning that when i produce a message, the Kafka that is up will receive the message

你们看到设置1有什么问题吗?

Do you guys see anything wrong with Set-up 1?

P.S如果您需要更多详细信息，我很乐意提供.

P.S If you need more details, i am happy to provide.

推荐答案

如果它可以帮助某人:

我有2个 Kafka 实例，但是我创建的任何主题的复制因子为1(由于我对其含义的误解/误解).

I had 2 Kafka instances, but my replication-factor for any topic created was 1(due to my misunderstanding/misinterpretation of its meaning).

这意味着，在创建主题时，将在 Kafka-1 或 Kafka-2 中创建主题，而不是同时在两个主题中创建主题.因此，当我尝试执行故障转移时，故障转移将失败，具体取决于我要写入的主题以及哪个 Kafka 被降级.

What this means is that, at the time of topic creation, the topic/s will be either created in Kafka-1 or Kafka-2, not both. As such, when i tried to do a fail-over, the fail-over will fail, depending on which topic i am writing to, and which Kafka was brought down.

简而言之，如果您有X个 Kafka 实例，则主题 replication-factor 必须为X( offsets.topic.replication.factor <也是如此/code>)

In short, if you have X Kafka instances, topic replication-factor needs to be X (Same goes for offsets.topic.replication.factor)

这篇关于测试“故障转移"功能.在卡夫卡的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！