在Kafka集群中添加新的ZooKeeper节点? [英] Adding new ZooKeeper node in Kafka cluster?

问题描述

我正在运行一个由五个节点组成的Apache Kafka集群，而我正在使用一个由三个节点组成的Apache ZooKeeper集群.

在zookeeper.properties文件中:

  server.1 = zNode01:2888:3888server.2 = zNode02:2888:3888server.3 = zNode03:2888:3888 

并在server.properties文件中:

  zookeeper.connect = zNode01:2181，zNode02:2181，zNode03:2181 

我想添加一个新的ZooKeeper节点:

1. 我需要将此新的ZooKeeper IP添加到现有的ZooKeeper属性文件中，并需要重新启动它，还是有另一种方法?

2. 我需要将此新的ZooKeeper IP添加到Kafka server.properties文件并需要重新启动它，还是有另一种方法?

解决方案

它比@ cricket_007所描述的要复杂得多.在您尝试将新成员添加到现有的Zookeeper集群之前，这将是一个很好的阅读.

https://zookeeper.apache.org/doc/r3.5.3-beta/zookeeperReconfig.html

重点放在修改当前动态配置" 部分.

基本上，这些是高级步骤:

a)必须将新服务器引入领导者.这是通过在Zookeeper.properties文件中添加自身和足够的群集信息"来完成的，以使连接器可以连接到现有的领导者.配置不需要绝对是最新的，而是足够新以与当前的领导者连接.为此，您只需从集群中的一个节点上获取zookeeper.properties文件，然后将联接器信息附加到该文件中，然后在联接器节点上启动zookeeper服务器.

b)请注意，木匠能够与集群的负责人交谈并不能使其自动成为集群的一部分.Zookeeper合奏必须投票并决定将新节点添加到群集中.联接程序的状态当前是一个无投票权的关注者，如果您查看zookeeper集成的当前配置(通过zkcli的"config"命令)，您将看不到该集成中列出的新节点.

c)现在，我们使用zkcli的"reconfig"命令以投票参与者或观察者的身份将新节点添加到集群中.投票参加者意味着所有共识性决定(例如，谁是新领导者，是否提交书面声明等)都将涉及所有投票参加者(而不是观察员).添加观察者主要是为了增加Zookeeper集合的读取吞吐量，而不会为每个写操作增加将它们包含在2相提交中的额外开销.reconfig命令还执行此两阶段提交，在此阶段中，领导者从所有投票参与者收集选票，而不管是否应将新节点添加到群集中.如果现有参与者的法定人数同意，则将新节点添加到群集中.

d)现在，执行zkcli的config命令会将新节点作为投票参与者或观察者显示为群集的一部分.

e)最后，您需要更新kafka的server.properties文件以关闭循环.即使可能不需要立即进行此更改，这也会通知kafka服务器(这是Zookeeper客户端)Zookeeper群集中新成员的可用性，以便在出现故障情况时可以回退到新添加的节点.

希望答案有助于理解如何将新节点动态添加到Zookeeper集群中.

I have running an Apache Kafka cluster of five nodes, and I am using an Apache ZooKeeper cluster of three nodes.

In zookeeper.properties file:

server.1=zNode01:2888:3888
server.2=zNode02:2888:3888
server.3=zNode03:2888:3888

And in server.properties file:

zookeeper.connect=zNode01:2181,zNode02:2181,zNode03:2181

I want to add a new ZooKeeper node:

1. I need to add this new ZooKeeper IP to an existing ZooKeeper properties file and need to restart it OR there is another way to do it?

2. I need to add this new ZooKeeper IP to Kafka server.properties file and need to restart it OR there is another way to do it?

解决方案

Its a more involved than what @cricket_007 described. This would be a good read before you attempt to add a new member to the existing zookeeper cluster.

https://zookeeper.apache.org/doc/r3.5.3-beta/zookeeperReconfig.html

Focus specifically on "Modifying the current dynamic configuration" section.

Basically, these are the high level steps:

a) The new server has to be introduced to the leader. This is done by adding itself and "enough cluster information" in the zookeeper.properties file for the joiner to connect to the existing leader. The configuration doesn't need to be absolutely uptodate, but fresh enough to connect with the current leader. To do that you could just get zookeeper.properties file from one of the nodes in the cluster, append joiner information to it, and start the zookeeper server on the joiner node.

b) Note that the joiner being able to talk to the leader of the cluster doesn't make it a part of the cluster automatically. The zookeeper ensemble has to vote and decide upon adding the new node into the cluster. The status of the joiner currently is a non-voting follower, and if you look at the current configuration of zookeeper ensemble (via zkcli's "config" command), you will not see the new node listed in the ensemble.

c) Now, we use the zkcli's "reconfig" command to add the new node to the cluster either as a voting participant or an observer. Voting participant means that all the consensus decisions (Eg. whos the new leader, whether to commit a write etc) will involve all the voting participants (and not the observers). Observers are added primarily to increase the read throughput of the zookeeper ensemble without adding the extra overhead of involving them in the 2-phase commit for each write operation. The reconfig command also performs this 2-phase commit, where the leader gathers votes from all the voting participants whether the new node should be added to the cluster. If quorum of the existing participants agree, the new node is added to the cluster.

d) Now, executing the zkcli's config command will show the new node as part of the cluster, either as a voting participant or as an observer.

e) Lastly, you would want to update the server.properties file of kafka to close the loop. Even though this change might not be immediately needed, this would inform kafka server (which is a zookeeper client) of the availability of the new member in the zookeeper cluster, so that it can fallback to the newly added node during failure scenarios.

Hope the answer helps in understanding how dynamically adding a new node to zookeeper cluster works.

这篇关于在Kafka集群中添加新的ZooKeeper节点?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！