如何在Kafka中实现强一致性? [英] How to achieve strong consistency in Kafka?

问题描述

尝试了解Kafka中的一致性维护.请找到方案并帮助理解.

Try to understanding consistency maintenance in Kafka. Please find the scenario and help to understand.

 Number of partition  = 2 
    Replication factor = 3 
    Number of broker in the cluster = 4 

在这种情况下，为了实现强一致性，应确认多少个节点. ack = all 或 ack = 3 或任何其他值.请确认相同.

In that case, for achieving the strong consistency how many nodes should acknowledge. Either ack = all or ack = 3 or any other value. Please confirm for the same.

推荐答案

您可能会对看到这是由Cloudera的工程师提供的，并且 Cloudera有他们的自己对Kafka可用性的说明

Which was given by an engineer at Cloudera, and Cloudera has their own documenation on Kafka availability

总而言之，一个以上的副本和高于1个的同步副本是一个好的开始.然后在生产者上，如果您可以牺牲吞吐量以提高数据可用性，这意味着您必须先写入所有副本，然后再继续操作，然后 acks = all .否则，如果您相信领导者经纪人在不洁的领导者选举中具有很高的可用性，那就是错误的，那么 acks = 1 在大多数情况下应该可以.

To summarize, more than 1 replica and higher than 1 in-sync replica is a good start. Then on the producer, if you are okay with sacrificing throughput for data availability, meaning you must have all replicas be written before continuing, then acks=all. Otherwise, if you trust the leader broker to be highly available with unclean leader election is false, then acks=1 should be okay in most cases.

acks = 3 不是有效的配置.我认为您正在寻找复制因子为 3 的 min.insync.replicas = 2 和 acks = all ；从上方链接

acks=3 isn't a valid config, by the way. I think you are looking for min.insync.replicas=2 and acks=all with a replication factor of 3; from above link

如果将 min.insync.replicas 设置为 2 ，并将 acks 设置为 all ，则每条消息必须成功地至少写入两个副本.这样可以确保除非两台主机都崩溃，否则消息不会丢失

If min.insync.replicas is set to 2 and acks is set to all, each message must be written successfully to at least two replicas. This guarantees that the message is not lost unless both hosts crash

此外，您还可以从Kafka 0.11开始使事务生成器能够朝着一次处理的方向工作

Also, you can enable the transactional producer, as of Kafka 0.11 to work towards exactly once processing

enable.idempotence=true

这篇关于如何在Kafka中实现强一致性?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！