如何决定Kafka集群大小 [英] How to decide Kafka Cluster size
问题描述
我打算决定 Kafka 集群上应该有多少个节点.我不确定要考虑的参数.我确定它必须 >=3(复制因子为 2，容错为 1 个节点).
I am planning to decide on how many nodes should be present on Kafka Cluster. I am not sure about the parameters to take into consideration. I am sure it has to be >=3 (with replication factor of 2 and failure tolerance of 1 node).
谁能告诉我在决定集群大小时应该记住哪些参数以及它们如何影响大小.
Can someone tell me what parameters should be kept in mind while deciding the cluster size and how they effect the size.
我知道以下因素，但不知道它如何在数量上影响集群大小.我知道它如何定性地影响集群大小.还有其他影响集群大小的参数吗?<代码>1.复制因子(集群大小>=复制因子)2.节点容错.(集群大小 >= 节点故障 + 1)
I know of following factors but don't know how it quantitatively effects the cluster size. I know how it qualitatively effect the cluster size. Is there any other parameter which effects cluster size?
1. Replication factor (cluster size >= replication factor)
2. Node failure tolerance. (cluster size >= node-failure + 1)
在考虑所有参数的情况下，以下场景的集群大小应该是多少<代码>1.有3个主题.2. 每个主题都有不同大小的消息.消息大小范围为 10 到 500kb.平均消息大小为 50kb.3. 每个主题有不同的分区.分区为 10、100、5004. 保留期为 7 天5. 每个主题每天发布 1 亿条消息.
What should be cluster size for following scenario while consideration of all the parameters
1. There are 3 topics.
2. Each topic has messages of different size. Message size range is 10 to 500kb. Average message size being 50kb.
3. Each topic has different partitions. Partitions are 10, 100, 500
4. Retention period is 7 days
5. There are 100 million messages which gets posted every day for each topic.
有人可以向我指出相关文档或任何其他可能讨论此问题的博客.我用谷歌搜索过，但无济于事
Can someone please point me to relevant documentation or any other blog which may discuss this. I have google searched it but to no avail
推荐答案
据我所知，从 Kafka 获得良好的吞吐量不仅仅取决于集群大小；还有其他配置需要考虑.我会尽量分享.
As I understand, getting good throughput from Kafka doesn't depend only on the cluster size; there are others configurations which need to be considered as well. I will try to share as much as I can.
Kafka 的吞吐量应该与您拥有的磁盘数量线性缩放.Kafka 0.8 中引入的新的多数据目录功能允许 Kafka 的主题在不同的机器上有不同的分区.随着分区数量的大幅增加，领导者选举过程变慢的可能性也会增加，这也会影响消费者的再平衡.这是一个需要考虑的问题，可能是一个瓶颈.
Kafka's throughput is supposed to be linearly scalabale with the numbers of disk you have. The new multiple data directories feature introduced in Kafka 0.8 allows Kafka's topics to have different partitions on different machines. As the partition number increases greatly, so do the chances that the leader election process will be slower, also effecting consumer rebalancing. This is something to consider, and could be a bottleneck.
另一个关键可能是磁盘刷新率.由于 Kafka 总是立即将所有数据写入文件系统，因此数据刷新到磁盘的频率越高，Kafka 的寻求绑定"就越多，吞吐量也就越低.同样，非常低的刷新率可能会导致不同的问题，因为在这种情况下要刷新的数据量会很大.所以提供一个确切的数字不太实际，我认为这就是你在 Kafka 文档中找不到这样直接的答案的原因.
Another key thing could be the disk flush rate. As Kafka always immediately writes all data to the filesystem, the more often data is flushed to disk, the more "seek-bound" Kafka will be, and the lower the throughput. Again a very low flush rate might lead to different problems, as in that case the amount of data to be flushed will be large. So providing an exact figure is not very practical and I think that is the reason you couldn't find such direct answer in the Kafka documentation.
还有其他因素.例如消费者的 fetch 大小、压缩、异步生产者的批量大小、套接字缓冲区大小等.
There will be other factors too. For example the consumer's fetch size, compressions, batch size for asynchronous producers, socket buffer sizes etc.
硬件&操作系统也将在这方面发挥关键作用，因为在基于 Linux 的环境中使用 Kafka 是可取的，因为它的 pageCache 机制用于将数据写入磁盘.在此处
Hardware & OS will also play a key role in this as using Kafka in a Linux based environment is advisable due to its pageCache mechanism for writing data to the disk. Read more on this here
您可能还想看看 操作系统刷新行为如何发挥关键作用 在您实际调整它以满足您的需要之前.我相信理解设计理念是关键，这使得它在吞吐量和容错方面非常有效.
You might also want to take a look at how OS flush behavior play a key role into consideration before you actually tune it to fit your needs. I believe it is key to understand the design philosophy, which makes it so effective in terms of throughput and fault-tolerance.
- https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines
- http://blog.liveramp.com/2013/04/08/kafka-0-8-producer-performance-2/
- https://grey-boundary.io/load-testing-apache-kafka-on-aws/
- https://cwiki.apache.org/confluence/display/KAFKA/性能+测试
这篇关于如何决定Kafka集群大小的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
