Kafka代替Zookeeper进行集群管理 [英] Kafka instead of Zookeeper for cluster management

问题描述

我正在编写一个位于Kafka之上的集群应用程序-它专门使用Kafka进行进程间通信和协调.我可以使用Zookeeper来管理集群-但是使用Kafka主题来管理集群并不是很困难.而且我考虑的更多，除了历史原因之外，似乎Kafka可以放弃Zookeeper并仅使用基于主题的解决方案

I am writing a clustered application sitting on top of Kafka -- it uses Kafka exclusively for interprocess communications and coordination. I could use Zookeeper to manage my cluster -- but it would not be very difficult to use Kafka topics to manage the cluster. And the more I think about it, other than for historical reasons, it seems like Kafka could drop Zookeeper and just use a topic-based solution

例如，在Kafka中可能有一个或多个特殊主题，您可以在其中发布Zookeeper当前跟踪的所有相同数据.经纪人，主题，分区，领导者等-似乎可以像通过Zookeeper一样容易地通过Kafka主题进行跟踪.

For example, there could be a special topic or topics in Kafka where you publish all of the same data currently kept track of in Zookeeper. Brokers, Topics, Partitions, Leaders, etc -- seems like this is just as easily tracked via Kafka topics as via Zookeeper.

我知道在Kafka 0.9.0中有一些远​​离Zookeeper的趋势，更多的是朝着此模型发展，并且记住我的问题不是关于Kafka的开发，而是更多我试图弄清楚应用程序的发展方向.

I know in Kafka 0.9.0 there's some movement away from Zookeeper, more towards this model, and remember my question is less about Kafka development or more me trying to figure out which direction to go in my application.

我不是要征求意见-我想知道的是Zookeeper提供的任何特定功能，如果使用基于Kafka/基于主题的协调方法，这些功能将很难实现.但是我什么也想不起来.

I'm not asking for an opinion -- what I want to know is are there any specific functions provided by Zookeeper that are going to be difficult with a Kafka/topic-based approach to coordination. But I can't think of anything.

即使是心跳监视(这也是我首先开始关注Zookeeper的原因)，您都可能会有一个客户端连接主题，客户端可以在加入集群时向其发布，以给定的时间间隔发布心跳，并在他们离开时发布.

Even heartbeat monitoring -- which was the reason I started looking at Zookeeper in the first place -- you could have a client connection topic, and clients could publish to it when they join the cluster, publish heartbeats at a given interval, and publish as they leave it.

推荐答案

让我们从一个以太空为视角的视图开始:您有两个分布式存储数据的系统.Zookeeper以某种形式将其数据组织在节点中目录的结构.Kafka将消息存储在主题内.

Let us start from a space eyed view: You have two distributed systems which store data. Zookeeper organizes it's data in nodes in some kind of directory like structure. Kafka stores messages within topics.

从鸟瞰角度来看，kafka是为实现高通量和可扩展性而打造的，而其中一位动物园管理员主要设计目标是一致性.Zookeeper旨在成为分布式应用程序，而Kafka可以看作是分布式提交日志.

From a bird eye view kafka is build for high-throughput and scalability while one of zookeepers main design goal is consistency. Zookeeper is mean to be a a Distributed Coordination Service for Distributed Applications while Kafka can be thought as a distributed commit log.

因此，您的问题的答案令人惊讶:取决于情况".为了协调我将使用zookeeper的分布式系统:那就是构建它的目的.你可以也可以使用kafka来执行此操作，但是有几件事需要完成如果您使用的是Zookeeper，则手册开箱即用.

So the answer to your question is surprisingly: 'It depends'. For coordinating a distributed system I would use zookeeper: Thats what it was build for. You could do this also with kafka but there are couple of things which needs to be done manualy which comes out of the box if you are using zookeeper.

一些例子:

• 一致性:ZK客户端可以选择是否需要强烈或最终的一致性
• 星历节点:与ZK-Watches一起对失败的服务做出反应是一件很棒的事情
• 顺序一致性:不授予您按照您向代理发送的顺序发送kafka消息的权限(仅授予对一部分中的消息进行排序的权限)
• ACL:从不使用它，但至少是kafka并未提供的现成的东西
• 序列节点

有关Zookeeper的功能，一个很好的概述是zookeeper-recipes:https://zookeeper.apache.org/doc/trunk/recipes.html

A pretty nice overview about what you can do with zookeeper are the zookeeper-recipes: https://zookeeper.apache.org/doc/trunk/recipes.html

:使用kafka对应用程序进行心跳检测当然是可能的.但是在我看来，动物园管理员中的短暂节点是更简单的选择.

: Heartbeating an application using kafka is of course possible. But ephemeral nodes in zookeeper are in my eyes the easier option.

这篇关于Kafka代替Zookeeper进行集群管理的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！