kafka MirrorMaker:使用者线程kafka-mirror不占用代理分区 [英] kafka MirrorMaker : No broker partitions consumed by consumer thread kafka-mirror

问题描述

这与kafka MirrorMaker工具有关.

我已经在两台计算机上配置了kafka.

1. 来源:

2. 目的地:vm [仅在源代码处使用ubuntu]

   源和目标上的卡夫卡都是相同版本的卡夫卡[kafka_2.11-0.9.0.0]

在源和目标上，各自的zookeeper和kafka服务器正在运行.

我想使用MirrorMaker工具从源到目标复制/制作主题的镜像.

下面是我使用过的命令:

 <代码> ./bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config ./config/mirror_consumer.properties --producer.config ./config/mirror_producer.properties --whitelist ='.*'&> mirror-log.log 

配置文件包含

  a.mirror_consumer.properties#host:要镜像的Kafka源Zookeeper端口zookeeper.connect =源IP:3181zookeeper.connection.timeout.ms = 1000000Consumer.timeout.ms = -1security.protocol = PLAINTEXTgroup.id = kafka-mirror在哪里，source-ip是源计算机的IP地址.我的来源的Zookeeper正在端口3181上运行.b.mirror_producer.properties#目的地的镜像代理(本地)bootstrap.servers =本地主机:9092producer.type =异步在哪里，本地主机，解析到目标即ubuntu vm并且kafka在默认端口即9092上运行 

最初，我创建了几个名为source1和source2的主题.在源计算机上，从命令行中有各自的生产者，我已经向创建的主题发送了一些消息.

从目标位置执行MirrorMaker命令后，

我可以看到目的地的消费者正在尝试消费主题.不幸的是，目的地的使用者无法从代理读取每个主题的分区.

请查看下面的示例日志条目:

  [2016-05-06 13:25:00,931]警告主题源1(kafka.consumer.RangeAssignor)的消费线程kafka-mirror_mojes-VirtualBox-1462521159741-6c2475c3-0没有消耗代理程序分区[2016-05-06 13:25:00,931]警告主题source1(kafka.consumer.RangeAssignor)的消费者线程kafka-mirror_mojes-VirtualBox-1462521295337-c3742307-0没有使用任何代理分区[2016-05-06 13:25:00,931]警告主题source2(kafka.consumer.RangeAssignor)的使用者线程kafka-mirror_mojes-VirtualBox-1462517840512-a134d048-0没有使用任何代理分区[2016-05-06 13:25:00,932]警告主题source2(kafka.consumer.RangeAssignor)的使用者线程kafka-mirror_mojes-VirtualBox-1462519206297-63bc9c58-0没有消耗任何代理分区[2016-05-06 13:25:00,932]警告主题source2(kafka.consumer.RangeAssignor)的使用者线程kafka-mirror_mojes-VirtualBox-1462519513695-bee7950e-0没有消耗代理程序分区 

如果您发现任何缺失/需要修复的东西，请告诉我.

那将是很大的帮助.

谢谢.

解决方案

当主题中的分区数量与提供给同一主题的消费者组中的消费者数量不匹配时，就会出现此问题./p>

This is regarding kafka MirrorMaker tool.

I have configured kafka on two machines.

1. source:

2. destination: vm [ubuntu at the source only]

   Kafka at both source and destination are of same version of kafka [kafka_2.11-0.9.0.0]

At source and destination, respective zookeeper and kafka servers are running.

with the MirrorMaker tool I wanted to replicate/make mirror of topics from source to destination.

Below is the command , that I have used:

./bin/kafka-run-class.sh kafka.tools.MirrorMaker --consumer.config ./config/mirror_consumer.properties --producer.config ./config/mirror_producer.properties --whitelist='.*' &>mirror-log.log

configuration files contains

a. mirror_consumer.properties

    #host:port of kafka source zookeeper to be mirrored
    zookeeper.connect=source-ip:3181
    zookeeper.connection.timeout.ms=1000000
    consumer.timeout.ms=-1
    security.protocol=PLAINTEXT
    group.id=kafka-mirror
 
    where, 

        source-ip is ip address of source machine.
        my zookeeper at source is running at port 3181.
   
b. mirror_producer.properties

    # mirror broker (local) at the destination
    bootstrap.servers=localhost:9092
    producer.type=async

   where,

       localhost, resolves to destination i.e. ubuntu vm
       and kafka is runnning on default port i.e. 9092

Initially, I have created few topics with name say source1 and source2. From source machine with respective producers from command line I have sent some messages to the topics created.

after executing the MirrorMaker command from destination,

I could see that the consumer at destination is trying to consume the topics. Unfortunately, consumer at destination fails to read the partitions from broker for each topic.

please have a look at the sample log entry below:

[2016-05-06 13:25:00,931] WARN No broker partitions consumed by consumer thread kafka-mirror_mojes-VirtualBox-1462521159741-6c2475c3-0 for topic source1 (kafka.consumer.RangeAssignor)
[2016-05-06 13:25:00,931] WARN No broker partitions consumed by consumer thread kafka-mirror_mojes-VirtualBox-1462521295337-c3742307-0 for topic source1 (kafka.consumer.RangeAssignor)
[2016-05-06 13:25:00,931] WARN No broker partitions consumed by consumer thread kafka-mirror_mojes-VirtualBox-1462517840512-a134d048-0 for topic source2 (kafka.consumer.RangeAssignor)
[2016-05-06 13:25:00,932] WARN No broker partitions consumed by consumer thread kafka-mirror_mojes-VirtualBox-1462519206297-63bc9c58-0 for topic source2 (kafka.consumer.RangeAssignor)
[2016-05-06 13:25:00,932] WARN No broker partitions consumed by consumer thread kafka-mirror_mojes-VirtualBox-1462519513695-bee7950e-0 for topic source2 (kafka.consumer.RangeAssignor)

Please let me know , if you see anything that is missing / need to be fixed.

It would be great help.

Thanks in advance.

解决方案

We get this issue when there is a mismatch between the number of partitions in a topic to the number of consumers in a consumer group feeding to the same topic.

这篇关于kafka MirrorMaker:使用者线程kafka-mirror不占用代理分区的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！