在 Kafka HA 中，为什么需要的最少代理数量是 3 而不是 2 [英] In Kafka HA, why minimum number of brokers required are 3 and not 2

问题描述

我们正在尝试使用 kafka 集群来实现 Kafka HA.在做研发的时候，我们发现zookeeper &推荐的最小节点数.kafka 经纪人是 3.

We are trying to implement Kafka HA using kafka cluster. While doing R&D, we found that minimum number of nodes recommended for zookeeper & kafka brokers are 3.

我们明白为什么zookeeper应该有最少3个节点，因为对于leader选举最少(n+1)/2个节点应该是up &运行.

We understand that why zookeeper should have minimum 3 nodes, because for leader election minimum (n+1)/2 nodes should be up & running.

但不清楚，为什么至少需要 3 个 kafka 代理.为什么我们不能用 2 个 kafka brokers & 来实现 HA3个zookeepr节点?

But its not clear, why minimum 3 kafka brokers are required. Why can't we implement HA with 2 kafka brokers & 3 zookeepr nodes?

推荐答案

Zookeeper 的最小节点数是 3，因为 quorum 属性.它应该是奇数，因为没有使用偶数个节点.例如:节点总数为 8 的 Zookeeper 可以降级为 7.由于共识算法，Zookeeper 中的许多节点也不好.(例如:Paxos)

The minimum number of nodes of Zookeeper is 3 because of the quorum attribute. It should be odd because the even number of nodes is no used. e.g: Zookeeper with total nodes of 8 can be downgraded to 7. Many nodes in Zookeepers also isn't good because of the consensus algorithm. (e.g: Paxos)

对于Kafka集群，个人认为设置2个broker是可以的.但最好有 3 个经纪人.原因是因为维护 ISR - In Sync Replicas.

For the Kafka cluster, personally I think it is okay for setting 2 brokers. But it is better with 3 brokers. The reason because of maintaining the ISR - In Sync Replicas.

假设您的 Kafka 集群有 2 个代理.为了保持数据的高可用性和一致性，我们将副本和 ISR 都设置为 2.有趣的部分是 min-ISR 属性.如果您将 min-ISR 设置为 1，则领导者失败，您可能没有任何剩余的副本.如果将 min-ISR 设置为 2，则当领导者或跟随者失败时，生产者和消费者都无法工作.

Let say your Kafka cluster has 2 brokers. To maintain the high availability and the consistency of the data, we will set the replicas and the ISR both to 2. The interesting part is the min-ISR attribute. If you set the min-ISR to 1 then the leader fails, likely you don't have any remaining replicas. If you set the min-ISR to 2, when either the leader or the follower fails, nor the producer and consumer can work.

如果我们的 Kafka 集群有 3 个 broker，并且我们将 ISR 设置为 3，则 min-ISR 等于 2.使用这种配置，我们接受在工作时丢失 1 个副本(领导者或追随者)的风险.例如，如果我们失去了领导者，那么至少有一个追随者同步切换.如果我们失去了一个追随者，我们仍然有一个剩余的追随者来保持 min-ISR 为 2.

If our Kafka cluster has 3 brokers and we set the ISR equals to 3, the min-ISR equals to 2. With this configuration, we accept the risk of losing 1 replica (either leader or follower) while working. For instance, if we lose the leader, there has at least one follower that in-sync for switching. If we lose one of the followers, we still have a remaining follower to keep the min-ISR to 2.

这篇关于在 Kafka HA 中，为什么需要的最少代理数量是 3 而不是 2的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！