kafka MirrorMaker : 消费者线程 kafka-mirror 没有使用代理分区 [英] kafka MirrorMaker : No broker partitions consumed by consumer thread kafka-mirror

问题描述

这是关于 kafka MirrorMaker 工具.

This is regarding kafka MirrorMaker tool.

我已经在两台机器上配置了 kafka.

I have configured kafka on two machines.

1. 来源:

目标:vm [仅限源代码的 ubuntu]

destination: vm [ubuntu at the source only]

源端和目的端的Kafka都是相同版本的kafka [kafka_2.11-0.9.0.0]

Kafka at both source and destination are of same version of kafka [kafka_2.11-0.9.0.0]

在源端和目标端，分别运行着 zookeeper 和 kafka 服务器.

At source and destination, respective zookeeper and kafka servers are running.

使用 MirrorMaker 工具，我想从源到目标复制/制作主题的镜像.

with the MirrorMaker tool I wanted to replicate/make mirror of topics from source to destination.

以下是我使用过的命令:

Below is the command , that I have used:

./bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config ./config/mirror_consumer.properties --producer.config ./config/mirror_producer.properties --whitelist='.*' &>mirror-log.log

配置文件包含

a. mirror_consumer.properties

    #host:port of kafka source zookeeper to be mirrored
    zookeeper.connect=source-ip:3181
    zookeeper.connection.timeout.ms=1000000
    consumer.timeout.ms=-1
    security.protocol=PLAINTEXT
    group.id=kafka-mirror
 
    where, 

        source-ip is ip address of source machine.
        my zookeeper at source is running at port 3181.
   
b. mirror_producer.properties

    # mirror broker (local) at the destination
    bootstrap.servers=localhost:9092
    producer.type=async

   where,

       localhost, resolves to destination i.e. ubuntu vm
       and kafka is runnning on default port i.e. 9092

最初，我创建了几个名为 source1 和 source2 的主题.从带有来自命令行的各个生产者的源机器上，我向创建的主题发送了一些消息.

Initially, I have created few topics with name say source1 and source2. From source machine with respective producers from command line I have sent some messages to the topics created.

从目的地执行 MirrorMaker 命令后，

after executing the MirrorMaker command from destination,

我可以看到目的地的消费者正在尝试使用这些主题.不幸的是，目的地的消费者无法从代理读取每个主题的分区.

I could see that the consumer at destination is trying to consume the topics. Unfortunately, consumer at destination fails to read the partitions from broker for each topic.

请查看下面的示例日志条目:

[2016-05-06 13:25:00,931] WARN No broker partitions consumed by consumer thread kafka-mirror_mojes-VirtualBox-1462521159741-6c2475c3-0 for topic source1 (kafka.consumer.RangeAssignor)
[2016-05-06 13:25:00,931] WARN No broker partitions consumed by consumer thread kafka-mirror_mojes-VirtualBox-1462521295337-c3742307-0 for topic source1 (kafka.consumer.RangeAssignor)
[2016-05-06 13:25:00,931] WARN No broker partitions consumed by consumer thread kafka-mirror_mojes-VirtualBox-1462517840512-a134d048-0 for topic source2 (kafka.consumer.RangeAssignor)
[2016-05-06 13:25:00,932] WARN No broker partitions consumed by consumer thread kafka-mirror_mojes-VirtualBox-1462519206297-63bc9c58-0 for topic source2 (kafka.consumer.RangeAssignor)
[2016-05-06 13:25:00,932] WARN No broker partitions consumed by consumer thread kafka-mirror_mojes-VirtualBox-1462519513695-bee7950e-0 for topic source2 (kafka.consumer.RangeAssignor)

如果您发现任何缺失/需要修复的内容，请告诉我.

Please let me know , if you see anything that is missing / need to be fixed.

这会很有帮助.

提前致谢.

推荐答案

当主题中的分区数量与提供给同一主题的消费者组中的消费者数量不匹配时，我们会遇到此问题.

We get this issue when there is a mismatch between the number of partitions in a topic to the number of consumers in a consumer group feeding to the same topic.

这篇关于kafka MirrorMaker : 消费者线程 kafka-mirror 没有使用代理分区的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！