如何在Kafka的镜像集群中维护客户补偿? [英] How customer offsets are maintained in mirrored cluster in Kafka?

问题描述

假设我有两个Kafka集群，并且正在使用镜像制造商将主题从一个群集镜像到另一个群集.我知道消费者有一个嵌入式生产者，可以向Kafka集群中的 __ consumer-offset 主题提交偏移量.我需要知道如果主要的Kafka群集出现故障会怎样?我们是否也同步 __ consumer-offset 主题?我认为，由于辅助群集可能具有不同数量的代理和其他设置.

Lets say I have two Kafka clusters and I am using mirror maker to mirror the topics from one cluster to another. I understand consumer has an embedded producer to commit offsets to __consumer-offset topic in Kafka cluster. I need to know what will happen if primary Kafka cluster goes down? Do we sync the __consumer-offset topic as well? Because secondary cluster could have different number of brokers and other settings, I think.

请告诉我们Kafka镜像集群如何处理消费者偏移?

Please tell how Kafka mirrored cluster takes care of consumer offset?

auto.offset.reset 设置是否在这里起作用?

Does auto.offset.reset setting play a role here?

推荐答案

更新

自Apache Kafka 2.7.0起，MirrorMaker能够复制已提交的偏移量.Cf https://cwiki.apache.org/confluence/display/KAFKA/KIP-545%3A+support+automated+consumer+offset+sync+across+clusters+in+MM+2.0

Since Apache Kafka 2.7.0, MirrorMaker is able to replicate committed offsets. Cf https://cwiki.apache.org/confluence/display/KAFKA/KIP-545%3A+support+automated+consumer+offset+sync+across+clusters+in+MM+2.0

原始答案

镜像制作者不复制偏移量.

Mirror maker does not replicate offsets.

此外， auto.offset.reset 与此完全无关，因为这是使用者设置，它定义了使用者应从何处开始读取案例，在启动时未找到有效的提交偏移量.

Furthermore, auto.offset.reset is completely unrelated to this, because it's a consumer setting that defines where a consumer should start reading for the case, that no valid committed offset is found at startup.

不镜像偏移量的原因基本上是因为它们不能在镜像群集上无意义，因为不能保证消息在两个群集中都具有相同的偏移量.

The reason for not mirroring offsets is basically, that they can be meaningless on the mirror cluster because it is not guaranteed, that the messages will have the same offsets in both cluster.

因此，在发生故障转移的情况下，您需要弄清楚聪明"的东西.自己一个人一种方法是记住上次处理记录的元数据时间戳.这使您可以寻找"目标.根据镜像群集上的时间戳在此处找到一个近似的偏移量.(您将需要使用Kafka 0.10.)

Thus, in fail over case, you need to figure out something "smart" by yourself. One way would be to remember the metadata timestamp of you last processed record. This allows you to "seek" based on timestamp on the mirror cluster to find an approximate offset there. (You will need Kafka 0.10 for this.)

这篇关于如何在Kafka的镜像集群中维护客户补偿?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！