为什么卡夫卡这么快 [英] Why Kafka so fast

问题描述

如果我有相同的硬件，使用 Kafka 或我们当前的解决方案(ServiceMix/Camel).有什么区别吗?Kafka 可以处理比它更大"的数据吗?为什么?

If I have same hardware, to use Kafka or our current solution(ServiceMix/Camel). Is there any difference? Can Kafka handle "bigger" data than it? Why?

有一篇文章说它可以有多快?但是我仍然不清楚为什么 Kafka 与其他解决方案相比如此之快?对 Apache Kafka 进行基准测试:200 万次写入每秒(在三台便宜的机器上)

There is a article to talk about how fast could it be? But I still don't get clearly why Kafka is so fast comparing to other solutions? Benchmarking Apache Kafka: 2 Million Writes Per Second (On Three Cheap Machines)

推荐答案

Kafka 运行速度快的原因有很多.仅举几例.

Kafka is fast for a number of reasons. To name a few.

• 零复制 - 请参阅https://en.wikipedia.org/wiki/Zero-copy 基本上它直接调用操作系统内核而不是在应用层移动数据快.
• 块中的批处理数据s - Kafka 就是要对数据进行批处理数据成块.这最大限度地减少了所有的跨机器延迟伴随于此的缓冲/复制.
• 避免随机磁盘访问 - 由于 Kafka 是一个不可变的提交日志，它不需要倒带磁盘和执行许多随机 I/O 操作，并且可以按顺序访问磁盘.这使它能够从一个物理磁盘与内存相比.
• 可以横向扩展 -能够为单个主题传播拥有数千个分区在数千台机器中意味着 Kafka 可以处理巨大的负载.

• Zero Copy - See https://en.wikipedia.org/wiki/Zero-copy basically it calls the OS kernal direct rather than at the application layer to move data fast.
• Batch Data in Chunks - Kafka is all about batching the data into chunks. This minimises cross machine latency with all the buffering/copying that accompanies this.
• Avoids Random Disk Access - as Kafka is an immutable commit log it does not need to rewind the disk and do many random I/O operations and can just access the disk in a sequential manner. This enables it to get similar speeds from a physical disk compared with memory.
• Can Scale Horizontally - The ability to have thousands of partitions for a single topic spread among thousands of machines means Kafka can handle huge loads.

这篇关于为什么卡夫卡这么快的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！