如何通过jmx监控kafka中的消费者滞后? [英] How to monitor consumer lag in kafka via jmx?

问题描述

我有一个 kafka 设置，其中包括一个到 prometheus 的 jmx 导出器.我正在寻找一个指标，它根据主题和 groupid 给出偏移滞后.我正在运行 kafka 2.2.0.

I have a kafka setup that includes a jmx exporter to prometheus. I'm looking for a metric, that gives the offset lag based on topic and groupid. I'm running kafka 2.2.0.

一些在线资源指向一个名为 kafka.consumer 的指标，但我的设置中没有这样的指标.

Some resources online point to a metric called kafka.consumer, but I have no such metric in my setup.

来自我的 jmx 终端:

From my jmxterminal:

$>domains
#following domains are available
JMImplementation
com.sun.management
java.lang
java.nio
java.util.logging
jdk.management.jfr
kafka
kafka.cluster
kafka.controller
kafka.coordinator.group
kafka.coordinator.transaction
kafka.log
kafka.network
kafka.server
kafka.utils

但是，我可以使用以下命令查看我需要的数据:

I am, however, able to see the data I need by using the following command:

root@kafka-0:/kafka# bin/kafka-consumer-groups.sh --describe --group benchmark_consumer_group --bootstrap-server localhost:9092
Consumer group 'benchmark_consumer_group' has no active members.

TOPIC               PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID     HOST            CLIENT-ID
benchmark_topic_10B 2          2795128         54223220        51428092        -               -               -
benchmark_topic_10B 9          4               4               0               -               -               -
benchmark_topic_10B 6          7               7               0               -               -               -
benchmark_topic_10B 7          5               5               0               -               -               -
benchmark_topic_10B 0          2834028         54224939        51390911        -               -               -
benchmark_topic_10B 1          15342331        54222342        38880011        -               -               -
benchmark_topic_10B 4          5               5               0               -               -               -
benchmark_topic_10B 5          6               6               0               -               -               -
benchmark_topic_10B 8          8               8               0               -               -               -
benchmark_topic_10B 3          4               4               0               -               -               -

但这无济于事，因为我需要根据指标进行跟踪.此外，此命令的执行时间约为 25 秒，因此用作指标来源不合理.

But that does not help since I need to track if from a metric. Also, this command takes about 25 seconds to execute, making it unreasonable to use as a source for metrics.

我的猜测是度量 kafka.consumer 在 2.2.0 版本中不存在，而是被另一个替换了.虽然，我无法在网上找到任何资源，其中包含有关如何以及从何处获取该指标的最新信息

My guess is that the metric kafka.consumer does not exist in version 2.2.0 and was replaced with another. Although, I can't find any resources online with up-to-date information on how and where to get that metric

推荐答案

kafka.consumer JMX 指标仅存在于使用者进程本身，而不存在于 Kafka 代理进程.请注意，您不会从使用 Java 以外的消费者库的消费者那里获得 kafka.consumer 指标.

The kafka.consumer JMX metrics are only present on the consumer processes themselves, not on the Kafka broker processes. Note that you would not get the kafka.consumer metric from consumers using a consumer library other than the Java one.

目前，Kafka 代理本身没有用于消费者延迟的可用 JMX 指标.还有其他通常用于监控消费者滞后的解决方案，例如 LinkedIn 的 Burrow.还有一些开源项目，例如 kafka9.offsets 通过 JMX 公开消费者滞后指标，但可能无法更新以使用最新的 Kafka.

Currently, there are no available JMX metrics for consumer lag from the Kafka broker itself. There are other solutions that are commonly used for monitoring consumer lag, such as Burrow by LinkedIn. There are also a few open source projects such as kafka9.offsets that expose consumer lag metrics via JMX, but may not be updated to work with the latest Kafka.

这篇关于如何通过jmx监控kafka中的消费者滞后?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！