卡夫卡经纪人自动缩放 [英] Kafka broker auto scaling

问题描述

我正在寻找有关Kafka经纪人根据负载自动缩放的一些建议.

I am looking for some suggestion on Kafka broker auto scaling up and down based on the load.

让我们说我们有一个电子商务网站，我们正在捕获某些活动或事件，并将这些事件发送到Kafka.由于在高峰时段，网站流量会更多，因此拥有固定数量的经纪人的理想kafka群集始终不是一个好主意，因此我们希望在网站流量更多时按比例增加经纪人的数量，并缩小规模流量较少时的经纪人数量.

Let us say we have an e-commerce site and we are capturing certain activities or events and these events are send to Kafka. Since during the peak hours/days the site traffic will be more so having the ideal kafka cluster with fixed number of brokers always is not a good idea so we want to scale it up the number of brokers when site traffic is more and scale it down the number of brokers when traffic is less.

人们如何解决此类问题?我无法在此主题中找到任何资源.任何帮助将不胜感激.

How does people solve this kind of issue? i am not able to find any resource in this topic. any help will be greatly appreciated.

推荐答案

Kafka并不是真的那样.从集群中添加/删除代理是一个非常实际的过程，它在集群上造成了很多额外的负载/开销，因此您不希望集群本身自动进行扩展或缩减.造成如此多额外开销的主要原因是，添加或删除代理需要在正常流量之上在整个集群中复制大量数据.基本上，来自失效代理的所有数据都需要复制到其他位置，以使主题/分区保持相同的复制因子，或者如果它是新代理，则需要将数据从其他代理重新混入其中，以便总体上减少了集群上的负载.复制所有这些数据会在群集上造成大量的IO/CPU负载，这可能足以引起严重的问题.

Kafka doesn't really work that way. Adding/removing brokers from the cluster is a very hands-on process, and it creates a lot of additional load/overhead on the cluster, so you wouldn't want the cluster to be automatically scaling up or down by itself. The main reason why it creates so much additional overhead is that adding or removing brokers requires lots of data copying across the cluster, on top of the normal traffic. Basically, all the data from a dead broker needs to be copied somewhere else, to keep the same replication factor for the topic/partitions, or if it's a new broker, data needs to be shuffled into it from the other brokers, so that the load on the cluster as a whole is reduced. All this data being copied around creates lots of IO/CPU load on the cluster, and it might be enough to cause significant problems.

处理这种情况的最佳方法是使用高峰时段期望流量的2倍甚至3倍进行性能测试和优化，并相应地构建集群.这样，如果突然出现尖峰，您将有足够的净空，而且您不必横向扩展/纵向扩展.

The best way to handle this scenario is to do performance testing and optimization with 2x or even 3x the traffic you'd expect during peak hours, and build out the cluster accordingly. This way, you'll have plenty of headroom if there are sudden spikes, and you won't have to scale-out/scale-in.

即使每秒处理数百万条消息，Kafka的性能也非常出色，因此您可能会发现应用程序/系统所需的群集大小并不像您最初想象的那么大/昂贵.

Kafka is extremely performant, even for traffic of millions of messages per second, so you will probably find that the cluster size your application/system requires is not as large/expensive as you initially thought.

这篇关于卡夫卡经纪人自动缩放的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！