kafka 和 JMX 导出器 [英] kafka and JMX-exporter

问题描述

我无法使用 JMX 导出器来公开 kafka 指标.你能看看我的步骤并在需要的地方纠正我吗?我正在按照步骤这里 使用 JMX 导出器启用 kafka.

I am unable to use JMX exporter to expose kafka metrics. Can you look at my steps and correct me where needed.? I am following steps here to enable kafka with JMX exporter.

以下是我遵循的分步说明

Following are step by step instruction I followed

#get kafka
wget kafka_2.11-2.0.0

# Download Prometheus JMX exporter:
sudo wget -P /opt/kafka/prometheus/ https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.3.0/jmx_prometheus_javaagent-0.3.0.jar
sudo wget -P /opt/kafka/prometheus/ https://raw.githubusercontent.com/prometheus/jmx_exporter/master/example_configs/kafka-0-8-2.yml

#Edit Prometheus JMX exporter config file; and append following lines 
echo  "- pattern : kafka.producer<type=producer-metrics, client-id=(.+)><>(.+):\w* name: kafka_producer_$2" >> /opt/kafka/prometheus/kafka-0-8-2.yml
echo  "— pattern : kafka.consumer<type=consumer-metrics, client-id=(.+)><>(.+):\w* name: kafka_consumer_$2" >> /opt/kafka/prometheus/kafka-0-8-2.yml
echo  "— pattern : kafka.consumer<type=consumer-fetch-manager-metrics, client-id=(.+)><>(.+):\w* name: kafka_consumer_$2" >> /opt/kafka/prometheus/kafka-0-8-2.yml


#start zookeeper in terminal 0
/opt/kafka/bin/zookeeper-server-start.sh config/zookeeper.properties


#start kafka broker in terminal 1
KAFKA_HEAP_OPTS=’"-Xmx1000M -Xms1000M"’ 
KAFKA_OPTS="-javaagent:/opt/kafka/prometheus/jmx_prometheus_javaagent-0.3.0.jar=7071:/opt/kafka/prometheus/kafka-0–8–2.yml" 
JMX_PORT=7071
/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties  


#start kafka consumer in terminal 2
KAFKA_OPTS="-javaagent:/opt/kafka/prometheus/jmx_prometheus_javaagent-0.3.0.jar=7072:/opt/kafka/prometheus/kafka-0–8–2.yml" 
JMX_PORT=7072
/opt/kafka/bin/kafka-console-consumer.sh — bootstrap-server 0.0.0.0:9092 — topic test — from-beginning

#start kafka producer in terminal 3
KAFKA_OPTS="-javaagent:/opt/kafka/prometheus/jmx_prometheus_javaagent-0.3.0.jar=7073:/opt/kafka/prometheus/kafka-0–8–2.yml" 
JMX_PORT=7073
/opt/kafka/bin/kafka-console-producer.sh — broker-list 0.0.0.0:9092 — topic test

经过上述步骤后，zookeeper 和 kafka 运行良好.我可以在生产者终端中输入一条消息，并在消费者控制台上收到它.但是，在 Prometheus 上看不到 kafka 指标.为了调试这个，我通过

After above steps zookeeper and kafka is running fine. I can type in producer terminal a message and it is received on consumer console. However no kafka metrics is visible on Prometheus. To debug this I checked ports 7071/2/3 by

netstat -tlnp | grep 7071
netstat -tlnp | grep 7072
netstat -tlnp | grep 7073

导致空白响应；这意味着没有服务使用上述端口.我觉得 JMX 导出器没有正确启用.

which resulted in blank response; this means no service is using above ports. I feel like JMX exporter is not enabled correctly.

你能帮我解决以上问题吗?

Can you help me with above issues?

推荐答案

从您的问题来看，您将变量放在自己的行上，而博客将它们放在同一行...

From the looks of your question, you put the variables on their own lines, while the blog has them on the same line...

例如这是如何启动Kafka服务器

e.g. This is how to start the Kafka server

KAFKA_HEAP_OPTS='"-Xmx1000M -Xms1000M"' KAFKA_OPTS='-javaagent:/opt/kafka/prometheus/jmx_prometheus_javaagent-0.3.0.jar=7071:/opt/kafka/prometheus-0–/ka8-2.yml' JMX_PORT=7081/opt/kafka/bin/kafka-server-start.sh/opt/kafka/config/server.properties

否则，您需要导出变量，以便子进程会像您一样提取它们在您之前的问题中，这似乎可以很好地公开指标

Otherwise, you need to export the variables so the sub-process will pick them up like you did in your previous question, which seemed to be working fine for exposing the metrics

export KAFKA_HEAP_OPTS='"-Xmx1000M -Xms1000M"'
export KAFKA_OPTS='-javaagent:/opt/kafka/prometheus/jmx_prometheus_javaagent-0.3.0.jar=7071:/opt/kafka/prometheus/kafka-0–8–2.yml'
export JMX_PORT=7081
/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties

注意:您链接的博客未使用 JMX_PORT，但这些端口不能与导出器相同.

Note: The blog you linked to doesn't use JMX_PORT, but those ports cannot be the same as the exporter.

我还建议至少下载一个高于 0.3 的版本 - https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/

I would also suggest at least downloading a version newer than 0.3 - https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/

并使用 Kafka 2.0 的配置 - https://github.com/prometheus/jmx_exporter/blob/master/example_configs/kafka-2_0_0.yml

and using the configs for Kafka 2.0 - https://github.com/prometheus/jmx_exporter/blob/master/example_configs/kafka-2_0_0.yml

旁注:netstat -tlnp |grep 707 会一次性显示所有内容

Sidenote: netstat -tlnp | grep 707 would show you all them at once

这篇关于kafka 和 JMX 导出器的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！