Kafka Mirror Maker:同步 __consumer_offsets 主题重复项 [英] Kafka Mirror Maker : Sync __consumer_offsets topic duplicates

问题描述

遵循此处提到的解决方案 kafka-mirror-maker-失败复制消费者偏移主题.我能够在 DC1(实时 Kafka 集群)和 DC2(备份 Kafka 集群)集群中启动镜像制造商而没有任何错误.

Following to the solution mentioned here kafka-mirror-maker-failing-to-replicate-consumer-offset-topic. I was able to start mirror maker without any error across DC1(Live Kafka cluster) and DC2(Backup Kafka cluster) clusters.

看起来它也可以跨 DC2 集群从 DC1 集群同步 __consumer_offsets 主题.

Looks like it is also able to sync __consumer_offsets topic across DC2 cluster form DC1 cluster.

问题

如果我关闭 DC1 的消费者并将相同的消费者(相同的 group_id)指向 DC2，即使镜像制造商能够同步此主题和分区的偏移量，它也会再次读取相同的消息.

If I close down consumer for DC1 and point same consumer(same group_id) to DC2 it reads the same messages again even though mirror maker is able sync offsets for this topic and partitions.

我可以看到 LOG-END-OFFSET 显示正确，但 CURRENT-OFFSET 仍然指向旧的导致 LAG.

I can see that LOG-END-OFFSET is showing correctly but CURRENT-OFFSET is still pointing to old causing LAG.

示例

• Mirror Maker 仍在 DC2 中运行.

• 消费者在 DC1 中关闭之前

• Mirror Maker is still running in DC2.

• Before consumer shut down in DC1

//DC1  __consumer_offsets topic
+-----------------------------------------------------------------+
| TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG  |
+-----------------------------------------------------------------+
| gs.suraj.test.1 0          10626           10626           0    |
| gs.suraj.test.1 2          10619           10619           0    |
| gs.suraj.test.1 1          10598           10598           0    |
+-----------------------------------------------------------------+

停止 DC1 中的消费者

Stop consumer in DC1

消费者在 DC2 中启动之前

Before consumer start up in DC2

//DC2  __consumer_offsets topic
+-----------------------------------------------------------------+
| TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG  |
+-----------------------------------------------------------------+
| gs.suraj.test.1 0          9098            10614           1516 |
| gs.suraj.test.1 2          9098            10614           1516 |
| gs.suraj.test.1 1          9098            10615           1517 |
+-----------------------------------------------------------------+

由于这种滞后，当我在 DC2 中启动同一个消费者时再次读取 4549 条消息，这不应该发生，因为它已经在 DC1 中读取了一个提交，并且镜像制造商已经从 DC1 同步 __consumer_offsets 主题到 DC2

Because of this lag, when I start same consumer in DC2 in reads 4549 messages again, which should not happen as it is already read an commited in DC1 and mirror maker have sync __consumer_offsets topic from DC1 to DC2

如果我在这里遗漏了什么，请告诉我.

Please let me know if I am missing anything in here.

推荐答案

如果您使用的是 Mirror Maker 2.0，他们明确说明不支持仅一次的动机:

If you are using Mirror Maker 2.0 they say explicitly on the motivation that there is no support for exactly-once:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0#KIP-382:MirrorMaker2.0-Motivation

但他们打算在未来这样做.

But they intend to do it in the future.

这篇关于Kafka Mirror Maker:同步 __consumer_offsets 主题重复项的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！