Kafka集群部署配置最佳实践? [英] Best Practices for Kafka Cluster Deployment Configuration?

问题描述

我在此要求提供一般最佳做法:

I'm asking for general best practices here:

如果我想要一个五节点集群，是否所有五个节点都运行包含 Zookeeper、Kafka、模式注册表的 Confluent Platform Umbrella 包?

If I want a five node cluster, do all five nodes run the Confluent Platform Umbrella Packages that include Zookeeper, Kafka, schema-registry?

是否曾建议在与 Kafka 集群不同的服务器上运行 Zookeper 集群?

Is it ever recommended to run the zookeper cluster on separate servers from the Kafka cluster?

如果我想运行 Kafka Connect 分布式工作者，我是否在所有集群节点上运行它?我曾经想在单独的服务器上运行吗?是推荐使用 Docker 还是不需要 Docker?

If I want to run the Kafka Connect distributed worker, do I run that on all cluster nodes? Do I ever want to run on separate servers? Is Docker recommended for this or is Docker unnecessary?

对于 Kafka Streaming 应用程序，它们是否应该在所有集群节点上运行?他们应该被码头化吗?它们应该在不同的节点上运行吗?

With Kafka Streaming apps, should they be run on all cluster nodes? Should they be dockerized? Should they ever run on separate nodes?

是否推荐使用 Mesos 之类的东西?

Is something like Mesos recommended?

推荐答案

在专用服务器(或虚拟服务器)上运行 Kafka Brokers 是最佳实践.Zookeeper 也是如此.

It is a best practice to run Kafka Brokers on dedicated servers (or virtual servers). The same is true of Zookeeper.

Confluent 平台的所有其他组件都可以在公共服务器或单独的机器上协同运行.

All the other components of the Confluent Platform can run colocated on common servers or on separate machines.

您通常只运行一个架构注册表(如果您想要容错，则可以运行两个).它们可以在任何可以连接回 Kafka Brokers 的机器上运行.

You would typically run only one Schema Registry (or two if you want fault tolerance). They can run on any machine that can connect back to the Kafka Brokers.

Kafka Connect 分布式工作者只需要在您想要托管 Kafka 连接器的机器上运行.他们只需要能够连接回 Kafka Brokers.

Kafka Connect distributed workers only need to run on machines that you want to host Kafka Connectors. They just need to be able to connect back to the Kafka Brokers.

Kafka Streams 应用程序可以在您想要的任何地方运行，只要它们可以连接回 Kafka Brokers.

Kafka Streams apps can run anywhere you want so long as they can connect back to the Kafka Brokers.

所有组件都可以在 docker 容器内运行，也可以在没有 docker 的情况下运行.

All components can run inside docker containers or without docker.

您可以使用任何您想要的微服务或数据中心资源管理工具(或根本不使用)——这是您的选择.

You can use whatever microservices or data center resource management tools you want (or none at all) - it is your choice.

这篇关于Kafka集群部署配置最佳实践?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！