如何通过JMX监控卡夫卡的消费者滞后? [英] How to monitor consumer lag in kafka via jmx?

问题描述

我有一个kafka设置，其中包括一个用于prometheus的jmx导出器.我正在寻找一个指标，该指标基于主题和groupid给出偏移量滞后.我正在运行kafka 2.2.0.

I have a kafka setup that includes a jmx exporter to prometheus. I'm looking for a metric, that gives the offset lag based on topic and groupid. I'm running kafka 2.2.0.

一些在线资源指向一个称为kafka.consumer的度量，但是我的设置中没有这样的度量.

Some resources online point to a metric called kafka.consumer, but I have no such metric in my setup.

从我的jmxterminal:

From my jmxterminal:

$>domains
#following domains are available
JMImplementation
com.sun.management
java.lang
java.nio
java.util.logging
jdk.management.jfr
kafka
kafka.cluster
kafka.controller
kafka.coordinator.group
kafka.coordinator.transaction
kafka.log
kafka.network
kafka.server
kafka.utils

但是，我可以使用以下命令查看所需的数据:

I am, however, able to see the data I need by using the following command:

root@kafka-0:/kafka# bin/kafka-consumer-groups.sh --describe --group benchmark_consumer_group --bootstrap-server localhost:9092
Consumer group 'benchmark_consumer_group' has no active members.

TOPIC               PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID     HOST            CLIENT-ID
benchmark_topic_10B 2          2795128         54223220        51428092        -               -               -
benchmark_topic_10B 9          4               4               0               -               -               -
benchmark_topic_10B 6          7               7               0               -               -               -
benchmark_topic_10B 7          5               5               0               -               -               -
benchmark_topic_10B 0          2834028         54224939        51390911        -               -               -
benchmark_topic_10B 1          15342331        54222342        38880011        -               -               -
benchmark_topic_10B 4          5               5               0               -               -               -
benchmark_topic_10B 5          6               6               0               -               -               -
benchmark_topic_10B 8          8               8               0               -               -               -
benchmark_topic_10B 3          4               4               0               -               -               -

但这无济于事，因为我需要从指标中进行跟踪.另外，此命令的执行时间大约为25秒，因此将其用作度量标准的来源是不合理的.

But that does not help since I need to track if from a metric. Also, this command takes about 25 seconds to execute, making it unreasonable to use as a source for metrics.

我的猜测是，度量标准kafka.consumer在版本2.2.0中不存在，并已被另一个替代.虽然，我无法在线找到任何资源以及有关如何以及在何处获取该指标的最新信息

My guess is that the metric kafka.consumer does not exist in version 2.2.0 and was replaced with another. Although, I can't find any resources online with up-to-date information on how and where to get that metric

推荐答案

kafka.consumer JMX指标仅出现在使用者进程本身上，而不出现在Kafka经纪人进程上.请注意，使用Java以外的使用者库，不会从使用者那里获得kafka.consumer指标.

The kafka.consumer JMX metrics are only present on the consumer processes themselves, not on the Kafka broker processes. Note that you would not get the kafka.consumer metric from consumers using a consumer library other than the Java one.

当前，没有来自Kafka经纪人本身的针对消费者滞后的JMX指标.还有其他通常用于监视消费者滞后的解决方案，例如LinkedIn的 Burrow .还有一些开源项目，例如 kafka9.offsets ，这些项目通过JMX公开了消费者滞后指标，但可能无法更新以与最新的Kafka一起使用.

Currently, there are no available JMX metrics for consumer lag from the Kafka broker itself. There are other solutions that are commonly used for monitoring consumer lag, such as Burrow by LinkedIn. There are also a few open source projects such as kafka9.offsets that expose consumer lag metrics via JMX, but may not be updated to work with the latest Kafka.

这篇关于如何通过JMX监控卡夫卡的消费者滞后?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！