手动设置Kafka消费者补偿 [英] Manually setting Kafka consumer offset

问题描述

在我们的项目中，有Active Kafka服务器(PR)和Passive Kafka服务器(DR)，两个Kafka代理都在我们的项目中配置了相同的组名，主题名和分区.从PR切换到DR时，已在DR上手动设置_consumer_offsets.

In our project, there are Active Kafka servers( PR) and Passive Kafka servers (DR), both Kafka brokers are configured with the same group name, topic name and partition in our project. When switching from PR to DR the _consumer_offsets is manually set on DR.

我在这里的问题是，Kafka使用者是否能够从上次读取的地方无缝使用消息?

My question here is, would the Kafka consumer be able to seamlessly consume the messages from where it was last read?

推荐答案

本身，依靠两个群集具有相同偏移量这一事实是错误的.

In itself, relying on the fact that both clusters will have the same offset is faulty.

偏移量-是相对特征.这不是消息的一部分.它实际上是文件内的位置.这些文件(Kafka日志文件)也会轮换并保留.不能保证这些日志文件在任何给定的时间点都是相同的.Kafka并没有声称要解决这样的问题.
此外，从CAP角度解决问题也很棘手.
除非您想要严格的物理复制，否则它也是毫无意义的.

Offset - is relative characteristic. It's not a part of a message. It's literally a position inside the file. And those files, Kafka log files, also rotate and have retentions. There's no guarantee that those log files are identical at any given point in time. Kafka doesn't claim to solve such an issue.
Besides, it's tricky to solve from CAP point of view.
And it's also pointless unless you want strict physical replication.

这就是为什么Kafka多集群工具通常是关于逻辑复制的原因.我没有使用过Mirror Maker( MM )，但是我已经使用了Replicator(这是Confluent的更高级的商业工具)，它具有一个称为的功能，谁会猜到，就像 MM 一种-偏移平移.复制器执行遵循:

That's why Kafka multi-cluster tools are usually about logical replication. I have not used Mirror Maker(MM) but I've used Replicator(which is a more advanced commercial tool by Confluent) and it has a feature for that called, who would have guessed, just like the MM one - offset translation. Replicator does the following:

• 从起源群集中的 __ consumer_timestamps 主题以了解消费者群体的进度.
• 翻译提交的偏移量从原始数据中心到目标中的相应偏移量数据中心.
• 将转换后的偏移量写入 __ consumer_offsets 只要在该目标群集中没有任何使用者，就可以在目标群集中找到该主题组已连接到目标群集.

• Reads the consumer offset and timestamp information from the __consumer_timestamps topic in the origin cluster to understand a consumer group’s progress.
• Translates the committed offsets in the origin datacenter to the corresponding offsets in the destination datacenter.
• Writes the translated offsets to the __consumer_offsets topic in the destination cluster, as long as no consumers in that group are connected to the destination cluster.

注意:您确实需要添加

Note: You do need to add an interceptor to your Kafka Consumers.

这篇关于手动设置Kafka消费者补偿的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！