磁盘持久性和复制性? [英] Disk persistence and replication?

问题描述

发送到Kafka的消息被写入磁盘并复制到容错.

Message sent to Kafka is written to disk and replicated for fault-tolerance.

我不确定何时将消息写入磁盘，为什么需要复制(分区)及其帮助?

I am not sure when message is written to disc, why replication(partition) is required and how it helps ?

据说Kafka的吞吐量很高，难道不影响磁盘性能吗?仅在内存中而不在磁盘中有消息是可以配置的吗?

Also Kafka is said to be high in throughput, Does not writing to disk impact performance ? Is it configurable to have message just in memory but not in disk ?

推荐答案

Jay Kreps(他是LinkedIn上的Kafka的原始架构师之一)的这篇博文解释了Kafka是如何作为提交日志进行独特设计的，可以在三台便宜的机器上每秒完成200万次写入"，并且产生的消息传递率比没有经过这种方式设计的传统消息代理要高得多.

This blog post by Jay Kreps (who is one of the original architects of Kafka while he was at LinkedIn) explains how Kafka is engineered uniquely as a commit log which can do "2 million writes per second on three cheap machines" and yield much higher messaging rates than traditional message brokers which are not engineered this way.

https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines

Kafka具有可配置的参数，用于控制将页面缓存刷新到磁盘的频率.不最终写入磁盘就无法运行Kafka.Kafka还将消息缓存在内存中以提高性能，因此它可以从内存中读取数据，从而提供高吞吐量，同时仍能保证持久性到磁盘以及集群复制数据的容错性，而不会造成消息丢失.

Kafka has configurable parameters that control how often page cache is flushed to disk. It is not possible to run Kafka without eventually writing to disk. Kafka also caches messages in memory for performance so it can give high throughput with reads from memory while still guaranteeing no message loss from persistence to disk and clustered replication of data for fault tolerance.

这篇关于磁盘持久性和复制性?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！