Kafka 代理自动扩展 [英] Kafka broker auto scaling

问题描述

我正在寻找一些关于 Kafka 代理根据负载自动伸缩的建议.

I am looking for some suggestion on Kafka broker auto scaling up and down based on the load.

假设我们有一个电子商务网站，我们正在捕获某些活动或事件，并将这些事件发送到 Kafka.由于在高峰时段/天，站点流量会更多，因此拥有具有固定数量代理的理想 kafka 集群始终不是一个好主意，因此我们希望在站点流量更多时扩大代理数量并缩小规模流量较少时的broker数量.

Let us say we have an e-commerce site and we are capturing certain activities or events and these events are send to Kafka. Since during the peak hours/days the site traffic will be more so having the ideal kafka cluster with fixed number of brokers always is not a good idea so we want to scale it up the number of brokers when site traffic is more and scale it down the number of brokers when traffic is less.

人们是如何解决这种问题的?我无法在此主题中找到任何资源.任何帮助将不胜感激.

How does people solve this kind of issue? i am not able to find any resource in this topic. any help will be greatly appreciated.

推荐答案

Kafka 并不是这样工作的.从集群中添加/删除代理是一个非常需要动手操作的过程，它会在集群上产生大量额外的负载/开销，因此您不希望集群自行自动扩展或缩减.它产生如此多额外开销的主要原因是，添加或删除代理需要在集群中进行大量数据复制，在正常流量之上.基本上，来自死代理的所有数据都需要复制到其他地方，以保持主题/分区的复制因子相同，或者如果它是新代理，则需要从其他代理将数据改组到其中，以便整个集群的负载减少了.所有这些被复制的数据都会在集群上造成大量 IO/CPU 负载，这可能足以导致重大问题.

Kafka doesn't really work that way. Adding/removing brokers from the cluster is a very hands-on process, and it creates a lot of additional load/overhead on the cluster, so you wouldn't want the cluster to be automatically scaling up or down by itself. The main reason why it creates so much additional overhead is that adding or removing brokers requires lots of data copying across the cluster, on top of the normal traffic. Basically, all the data from a dead broker needs to be copied somewhere else, to keep the same replication factor for the topic/partitions, or if it's a new broker, data needs to be shuffled into it from the other brokers, so that the load on the cluster as a whole is reduced. All this data being copied around creates lots of IO/CPU load on the cluster, and it might be enough to cause significant problems.

处理这种情况的最佳方法是在高峰时段使用 2 倍甚至 3 倍的流量进行性能测试和优化，并相应地构建集群.这样，如果突然出现峰值，您将有足够的空间，并且您不必横向扩展/缩减.

The best way to handle this scenario is to do performance testing and optimization with 2x or even 3x the traffic you'd expect during peak hours, and build out the cluster accordingly. This way, you'll have plenty of headroom if there are sudden spikes, and you won't have to scale-out/scale-in.

Kafka 的性能非常出色，即使是每秒数百万条消息的流量，因此您可能会发现您的应用程序/系统所需的集群规模并不像您最初想象的那么大/昂贵.

Kafka is extremely performant, even for traffic of millions of messages per second, so you will probably find that the cluster size your application/system requires is not as large/expensive as you initially thought.

这篇关于Kafka 代理自动扩展的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！