使用 KafkaOffsetMonitoring 工具监控 Kafka Spout [英] Monitoring Kafka Spout with KafkaOffsetMonitoring tool

问题描述

我正在为我的项目使用 Storm-0.9.2 发行版附带的 kafkaSpout.我想监控这个 spout 的吞吐量.我尝试使用 KafkaOffsetMonitoring，但它没有显示任何消费者阅读我的主题.

I am using the kafkaSpout that came with storm-0.9.2 distribution for my project. I want to monitor the throughput of this spout. I tried using the KafkaOffsetMonitoring, but it does not show any consumers reading from my topic.

我怀疑这是因为我在 Zookeeper 中为 spout 指定了根路径来存储消费者偏移量.kafkaOffsetMonitor 如何知道在哪里查找有关我的 kafkaSpout 实例的数据?

I suspect this is because I have specified the root path in Zookeeper for the spout to store the consumer offsets. How will the kafkaOffsetMonitor know that where to look for data about my kafkaSpout instance?

有人能解释一下zookeeper究竟在哪里存储关于kafka主题和消费者的数据吗?zookeeper 是一个文件系统.那么，它是如何排列不同topic及其分区的数据的呢?什么是消费者 groupid，在存储消费者偏移量时，zookeeper 是如何解释的?

Can someone explain exactly where does zookeeper store data about kafka topics and consumers? The zookeeper is a filesystem. So, how does it arrange data of different topics and their partitions? What is consumer groupid and how is it interpreted by zookeeper while storing consumer offset?

如果有人使用过 kafkaOffsetMonitor 来监控 kafkaSpout 的吞吐量，请告诉我如何使用该工具找到我的 spout?

If anyone has ever used kafkaOffsetMonitor to monitor throughput of a kafkaSpout, please tell me how I can get the tool to find my spout?

非常感谢，帕拉克沙

推荐答案

我没用过KafkaOffsetMonitor，但我可以回答另一部分.

I have never used KafkaOffsetMonitor, but I can answer the other part.

zookeeper.connect 是您可以为 Kafka 指定 znode 的属性；默认情况下，它会将所有数据保留在/".

zookeeper.connect is the property where you can specify the znode for Kafka; By default it keeps all data at '/'.

您可以使用 zkCli.sh(zookeeper 命令行)访问 zookeeper 文件系统.您应该查看/consumers 和/brokers；以下会给你偏移量

You can access the zookeeper filesystem using zkCli.sh, the zookeeper command line. You should look at /consumers and /brokers; following would give you the offset

get /consumers/my_test_group/offsets/my_topic/0

您可以连续轮询此偏移量以了解 spout 的消耗率.

You can poll this offset continuously to know the rate of consumption at spout.

这篇关于使用 KafkaOffsetMonitoring 工具监控 Kafka Spout的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！