Kafka 镜像集群中如何维护客户偏移量? [英] How customer offsets are maintained in mirrored cluster in Kafka?

问题描述

假设我有两个 Kafka 集群，我正在使用 mirror maker 将主题从一个集群镜像到另一个集群.我知道消费者有一个嵌入式生产者来向 Kafka 集群中的 __consumer-offset 主题提交偏移量.我需要知道如果主 Kafka 集群出现故障会发生什么?我们是否也同步 __consumer-offset 主题?因为辅助集群可能有不同数量的代理和其他设置，我认为.

Lets say I have two Kafka clusters and I am using mirror maker to mirror the topics from one cluster to another. I understand consumer has an embedded producer to commit offsets to __consumer-offset topic in Kafka cluster. I need to know what will happen if primary Kafka cluster goes down? Do we sync the __consumer-offset topic as well? Because secondary cluster could have different number of brokers and other settings, I think.

请问Kafka镜像集群如何处理consumer offset?

Please tell how Kafka mirrored cluster takes care of consumer offset?

auto.offset.reset 设置在这里起作用吗?

Does auto.offset.reset setting play a role here?

推荐答案

更新

从 Apache Kafka 2.7.0 开始，MirrorMaker 能够复制提交的偏移量.参见 https://cwiki.apache.org/confluence/display/KAFKA/KIP-545%3A+support+automated+consumer+offset+sync+across+clusters+in+MM+2.0

Since Apache Kafka 2.7.0, MirrorMaker is able to replicate committed offsets. Cf https://cwiki.apache.org/confluence/display/KAFKA/KIP-545%3A+support+automated+consumer+offset+sync+across+clusters+in+MM+2.0

原答案

镜像制造商不复制偏移量.

Mirror maker does not replicate offsets.

此外，auto.offset.reset 与此完全无关，因为它是一个消费者设置，它定义了消费者应该从哪里开始读取案例，在启动时找不到有效的提交偏移量.

Furthermore, auto.offset.reset is completely unrelated to this, because it's a consumer setting that defines where a consumer should start reading for the case, that no valid committed offset is found at startup.

不镜像偏移的原因基本上是，它们在镜像集群上可能毫无意义，因为不能保证消息在两个集群中具有相同的偏移.

The reason for not mirroring offsets is basically, that they can be meaningless on the mirror cluster because it is not guaranteed, that the messages will have the same offsets in both cluster.

因此，在故障转移的情况下，您需要找出一些聪明"的方法；由你自己.一种方法是记住上次处理记录的元数据时间戳.这使您可以寻找"根据镜像集群上的时间戳找到那里的近似偏移量.(为此您需要 Kafka 0.10.)

Thus, in fail over case, you need to figure out something "smart" by yourself. One way would be to remember the metadata timestamp of you last processed record. This allows you to "seek" based on timestamp on the mirror cluster to find an approximate offset there. (You will need Kafka 0.10 for this.)

这篇关于Kafka 镜像集群中如何维护客户偏移量?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！