Kafka MirrorMaker2 自动消费者偏移同步 [英] Kafka MirrorMaker2 automated consumer offset sync

问题描述

我正在使用 MirrorMaker2 进行 DR.

Kafka 2.7 应该支持

所有源集群主题都复制到目标集群中.另外 ...checkpoint.internal 主题是在目标集群中创建的，其中包含同步的所有源集群偏移量，但我没有看到这些偏移量被转换为目标集群 _consumer_offsets 主题这意味着当我在目标集群中启动消费者(同一个消费者组)时，它将从头开始读取消息.

我的期望是，在允许自动消费者偏移量后，同步所有来自源集群的消费者偏移量，并转换并存储在目标集群的 _consumer_offsets 主题中.

有人可以澄清我的期望是否正确，如果不正确，它应该如何工作.

解决方案

sync.group.offsets.enabled 设置用于 MirrorCheckpointConnector.

我不完全确定 Strimzi 如何运行 MirrorMaker 2，但我认为您需要将其设置为:

checkpointConnector:配置:checkpoints.topic.replication.factor: 1sync.group.offsets.enabled:真"；

I am using MirrorMaker2 for DR.

Kafka 2.7 should support automated consumer offset sync

Here is the yaml file I am using (I use strimzi for creating it)

All source cluster topics are replicated in destination cluster. Also ...checkpoint.internal topic is created in destination cluster that contains all source cluster offsets synced, BUT I don't see these offsets being translated into destination cluster _consumer_offsets topic which means when I will start consumer (same consumer group) in destination cluster it will start reading messages from the beginning.

My expectation is that after allowing automated consumer offsets sync all consumer offsets from source clusters translated and stored in _consumer_offsets topic in the destination cluster.

Can someone please clarify if my expectation is correct and if not how it should work.

解决方案

The sync.group.offsets.enabled setting is for MirrorCheckpointConnector.

I'm not entirely sure how Strimzi runs MirrorMaker 2 but I think you need to set it like:

checkpointConnector:
  config:
    checkpoints.topic.replication.factor: 1
    sync.group.offsets.enabled: "true"

这篇关于Kafka MirrorMaker2 自动消费者偏移同步的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！