Kafka 代替 Zookeeper 进行集群管理 [英] Kafka instead of Zookeeper for cluster management

问题描述

我正在编写一个基于 Kafka 的集群应用程序——它专门使用 Kafka 进行进程间通信和协调.我可以使用 Zookeeper 来管理我的集群——但是使用 Kafka 主题来管理集群并不是很困难.而且我想得越多，除了历史原因，似乎Kafka可以放弃Zookeeper，只使用基于主题的解决方案

I am writing a clustered application sitting on top of Kafka -- it uses Kafka exclusively for interprocess communications and coordination. I could use Zookeeper to manage my cluster -- but it would not be very difficult to use Kafka topics to manage the cluster. And the more I think about it, other than for historical reasons, it seems like Kafka could drop Zookeeper and just use a topic-based solution

例如，Kafka 中可能有一个或多个特殊主题，您可以在其中发布当前在 Zookeeper 中跟踪的所有相同数据.Brokers、Topics、Partitions、Leaders 等等——似乎通过 Kafka 主题和通过 Zookeeper 一样容易跟踪.

For example, there could be a special topic or topics in Kafka where you publish all of the same data currently kept track of in Zookeeper. Brokers, Topics, Partitions, Leaders, etc -- seems like this is just as easily tracked via Kafka topics as via Zookeeper.

我知道在 Kafka 0.9.0 中，有一些远离 Zookeeper 的移动，更多地转向这个模型，并且记住我的问题不是关于 Kafka 开发，而是更多我试图找出我的应用程序的方向.

I know in Kafka 0.9.0 there's some movement away from Zookeeper, more towards this model, and remember my question is less about Kafka development or more me trying to figure out which direction to go in my application.

我不是在征求意见——我想知道的是，Zookeeper 提供的任何特定功能在使用基于 Kafka/主题的协调方法时会很困难.但我什么也想不起来.

I'm not asking for an opinion -- what I want to know is are there any specific functions provided by Zookeeper that are going to be difficult with a Kafka/topic-based approach to coordination. But I can't think of anything.

即使是心跳监控——这也是我最初开始研究 Zookeeper 的原因——你可以有一个客户端连接主题，客户端可以在加入集群时发布到它，在给定的时间间隔发布心跳，并在他们离开时发布.

Even heartbeat monitoring -- which was the reason I started looking at Zookeeper in the first place -- you could have a client connection topic, and clients could publish to it when they join the cluster, publish heartbeats at a given interval, and publish as they leave it.

推荐答案

让我们从空间视角开始:你有两个分布式存储数据的系统.Zookeeper 以某种方式在节点中组织它的数据目录结构.Kafka 在主题中存储消息.

Let us start from a space eyed view: You have two distributed systems which store data. Zookeeper organizes it's data in nodes in some kind of directory like structure. Kafka stores messages within topics.

从鸟瞰角度来看，kafka 是为高吞吐量和可扩展性而构建的，而动物园管理员之一主要设计目标是一致性.Zookeeper 是一个分布式协调服务分布式应用程序，而 Kafka 可以被认为是分布式提交日志.

From a bird eye view kafka is build for high-throughput and scalability while one of zookeepers main design goal is consistency. Zookeeper is mean to be a a Distributed Coordination Service for Distributed Applications while Kafka can be thought as a distributed commit log.

因此，您的问题的答案令人惊讶:这取决于".用于协调我会使用zookeeper的分布式系统:这就是它的构建目的.你可以也用 kafka 做这件事，但有几件事需要做如果您使用的是zookeeper，它是开箱即用的.

So the answer to your question is surprisingly: 'It depends'. For coordinating a distributed system I would use zookeeper: Thats what it was build for. You could do this also with kafka but there are couple of things which needs to be done manualy which comes out of the box if you are using zookeeper.

一些例子:

• 一致性:ZK-Client 可以选择是否需要强一致性或最终一致性
• 临时节点:与 ZK-Watches 一起是对失败服务做出反应的好方法
• 顺序一致性:不能按照您将其写入代理的顺序接收 kafka 消息(仅允许对分区内的消息进行排序)
• ACL:从未使用过它，但至少是 kafka 没有提供的东西
• 序列节点

关于你可以用 zookeeper 做什么的一个很好的概述是 zookeeper-recipes:https://zookeeper.apache.org/doc/trunk/recipes.html

A pretty nice overview about what you can do with zookeeper are the zookeeper-recipes: https://zookeeper.apache.org/doc/trunk/recipes.html

:当然可以使用 kafka 对应用程序进行心跳检测.但是在我看来，zookeeper 中的临时节点是更简单的选择.

: Heartbeating an application using kafka is of course possible. But ephemeral nodes in zookeeper are in my eyes the easier option.

这篇关于Kafka 代替 Zookeeper 进行集群管理的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！