磁盘持久化和复制? [英] Disk persistence and replication?

问题描述

发送到 Kafka 的消息被写入磁盘并复制容错.

Message sent to Kafka is written to disk and replicated for fault-tolerance.

我不确定何时将消息写入磁盘，为什么需要复制(分区)以及它如何提供帮助?

I am not sure when message is written to disc, why replication(partition) is required and how it helps ?

据说Kafka的吞吐量也很高，写入磁盘不会影响性能吗?是否可以将消息配置为仅在内存中而不在磁盘中?

Also Kafka is said to be high in throughput, Does not writing to disk impact performance ? Is it configurable to have message just in memory but not in disk ?

推荐答案

Jay Kreps(他在 LinkedIn 时是 Kafka 的原始架构师之一)的这篇博文解释了 Kafka 如何独特地设计为提交日志可以在三台便宜的机器上每秒进行 200 万次写入"，并且比非以这种方式设计的传统消息代理产生更高的消息传递率.

This blog post by Jay Kreps (who is one of the original architects of Kafka while he was at LinkedIn) explains how Kafka is engineered uniquely as a commit log which can do "2 million writes per second on three cheap machines" and yield much higher messaging rates than traditional message brokers which are not engineered this way.

https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines

Kafka 有可配置的参数来控制页面缓存刷新到磁盘的频率.如果不最终写入磁盘，就不可能运行 Kafka.Kafka 还将消息缓存在内存中以提高性能，因此它可以通过从内存读取来提供高吞吐量，同时仍然保证从持久性到磁盘和集群复制数据不会丢失消息以实现容错.

Kafka has configurable parameters that control how often page cache is flushed to disk. It is not possible to run Kafka without eventually writing to disk. Kafka also caches messages in memory for performance so it can give high throughput with reads from memory while still guaranteeing no message loss from persistence to disk and clustered replication of data for fault tolerance.

这篇关于磁盘持久化和复制?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！