使用KafkaOffsetMonitoring工具监视Kafka Spout [英] Monitoring Kafka Spout with KafkaOffsetMonitoring tool

问题描述

我正在为我的项目使用Storm-0.9.2发行版随附的kafkaSpout.我要监视此喷嘴的吞吐量.我尝试使用KafkaOffsetMonitoring，但是它没有显示任何消费者在阅读我的主题.

I am using the kafkaSpout that came with storm-0.9.2 distribution for my project. I want to monitor the throughput of this spout. I tried using the KafkaOffsetMonitoring, but it does not show any consumers reading from my topic.

我怀疑这是因为我已经在Zookeeper中为喷口指定了根路径，以存储使用者偏移量. kafkaOffsetMonitor如何知道在哪里查找有关我的kafkaSpout实例的数据?

I suspect this is because I have specified the root path in Zookeeper for the spout to store the consumer offsets. How will the kafkaOffsetMonitor know that where to look for data about my kafkaSpout instance?

有人可以确切解释动物园管理员在哪里存储有关kafka主题和消费者的数据吗? Zookeeper是一个文件系统.那么，它如何排列不同主题及其分区的数据?什么是消费者组标识?在存储消费者偏移量时，动物园管理员如何解释它?

Can someone explain exactly where does zookeeper store data about kafka topics and consumers? The zookeeper is a filesystem. So, how does it arrange data of different topics and their partitions? What is consumer groupid and how is it interpreted by zookeeper while storing consumer offset?

如果有人曾经使用kafkaOffsetMonitor来监视kafkaSpout的吞吐量，请告诉我如何获得该工具以找到我的壶嘴?

If anyone has ever used kafkaOffsetMonitor to monitor throughput of a kafkaSpout, please tell me how I can get the tool to find my spout?

非常感谢， 沙捞越

推荐答案

我从未使用过KafkaOffsetMonitor，但是我可以回答其他部分.

I have never used KafkaOffsetMonitor, but I can answer the other part.

zookeeper.connect 是可以为Kafka指定znode的属性.默认情况下，它将所有数据保留在'/'.

zookeeper.connect is the property where you can specify the znode for Kafka; By default it keeps all data at '/'.

您可以使用zookeeper命令行zkCli.sh访问zookeeper文件系统. 您应该查看/consumers和/brokers；以下将为您提供偏移量

You can access the zookeeper filesystem using zkCli.sh, the zookeeper command line. You should look at /consumers and /brokers; following would give you the offset

get /consumers/my_test_group/offsets/my_topic/0

您可以连续轮询此偏移量，以了解喷口处的消耗率.

You can poll this offset continuously to know the rate of consumption at spout.

这篇关于使用KafkaOffsetMonitoring工具监视Kafka Spout的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！