从Confluent Replicator迁移到Apache Mirror Maker 2.0时保持相同的偏移量 [英] Maintaining same offset while migrating from Confluent Replicator to Apache Mirror Maker 2.0

问题描述

我们当前正在尝试将Confluent复制器迁移到Apache Open Source Mirror Maker v2.0.我们面临的一个问题是，当镜像制造商在同一主题上启动时，复制器已经复制的消息将再次被复制.这不会发生，因为消息在目标群集处变得重复.这里是更多详细信息:

We are currently trying to migrate Confluent replicator to Apache Open Source Mirror Maker v2.0. We are facing an issue where the messages which are already replicated by replicator is getting replicated again when the mirror maker is started on the same topic. This should not happen as messages are getting duplicated at the target cluster. Here are more details:

1. RCA:复制器分配用于复制消息的使用者组.该使用者组维护源主题的偏移量.但是我们无法将相同的使用者组分配给镜像制造商2中的使用者配置.
2. Mirror Maker 1.0:可以在consumer.properties文件中分配作为同一使用者组的工作，并在复制器停止后立即拾取消息.
3. 尝试在所有可用选项(群集模式，独立连接和分布式连接模式)中在Mirror Maker 2.0中运行和配置 source.cluster.consumer.group.id 复制邮件时，正在将使用者组ID分配为空.

1. RCA: replicator assign a consumer group for replicating messages. This consumer group maintains the offset of the source topic. But we are not able to assign same consumer group to the Consumer config in mirror maker 2.
2. Mirror Maker 1.0 : working as same consumer group can be assigned in consumer.properties file and the messages are picked right after where replicator was stopped.
3. Tried running and configuring source.cluster.consumer.group.id in mirror maker 2.0 in all available options (in cluster mode, in connect-standalone and connect-distributed mode) but mirror maker 2.0 is assigning consumer group id as null while replicating messages.

任何指针(如果有人做了相同的事情，并尝试与mirror maker 2.0保持相同的偏移量.

Any pointers if anyone has done same and tried to maintain the same offset with mirror maker 2.0.

推荐答案

我们获得了解决此问题的粗略方法.以下是高级步骤:

We got a crude way to solve this issue. Below are the high level steps:

• 从Replicator的内部主题中读取消息以存储偏移量.[连接偏移]
• 该主题存储将在key:value对中复制的所有主题的偏移量.例如

• Read the message from Replicator's internal topic for storing offsets. [connect-offsets]
• This topic stores the offsets for all topics which is getting replicated in key:value pair . For e.g

键:[复制器组"，{主题":测试"，分区":0}]
值:{"offset":24}

Key : ["replicator-group",{"topic":"TEST","partition":0}]
Value: {"offset":24}

• 对于每个主题和分区，每当复制新消息时，都会使用相同的键复制一条新消息，但是会增加connect-offsets主题的偏移量.
• 将此消息的密钥转换为Mirror Maker 2格式，并在MirrorMaker2的内部主题中生成它.[您可以在mirrormaker2-connect-distributed.properties文件中更改内部主题.镜像制造商内部主题的格式为:

键:[镜像制造者组"，{集群" ::"，分区":0，主题":测试"}]
值:{"offset":24}

Key: ["mirrormaker-group",{"cluster":"","partition":0,"topic":"TEST"}]
Value: {"offset":24}

• 发布消息后，镜像制造商重新启动后，它将读取内部主题以获取该主题的最新偏移量，必须为其复制消息，并且通过这种方式，我们可以确保没有重复的消息被复制./li>
  • After posting the message, once the mirror maker is restarted, it will read the internal topic to get the latest offset of that topic for which the message has to be replicated and this way we can ensure no duplicate messages are replicated.

  这篇关于从Confluent Replicator迁移到Apache Mirror Maker 2.0时保持相同的偏移量的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！