卡夫卡和JMX出口商 [英] kafka and JMX-exporter

问题描述

我无法使用JMX导出器公开kafka指标.您能看看我的脚步，并在需要时纠正我吗? 我正在按照步骤此处以使用JMX导出器启用kafka.

I am unable to use JMX exporter to expose kafka metrics. Can you look at my steps and correct me where needed.? I am following steps here to enable kafka with JMX exporter.

以下是我遵循的分步说明

Following are step by step instruction I followed

#get kafka
wget kafka_2.11-2.0.0

# Download Prometheus JMX exporter:
sudo wget -P /opt/kafka/prometheus/ https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.3.0/jmx_prometheus_javaagent-0.3.0.jar
sudo wget -P /opt/kafka/prometheus/ https://raw.githubusercontent.com/prometheus/jmx_exporter/master/example_configs/kafka-0-8-2.yml

#Edit Prometheus JMX exporter config file; and append following lines 
echo  "- pattern : kafka.producer<type=producer-metrics, client-id=(.+)><>(.+):\w* name: kafka_producer_$2" >> /opt/kafka/prometheus/kafka-0-8-2.yml
echo  "— pattern : kafka.consumer<type=consumer-metrics, client-id=(.+)><>(.+):\w* name: kafka_consumer_$2" >> /opt/kafka/prometheus/kafka-0-8-2.yml
echo  "— pattern : kafka.consumer<type=consumer-fetch-manager-metrics, client-id=(.+)><>(.+):\w* name: kafka_consumer_$2" >> /opt/kafka/prometheus/kafka-0-8-2.yml


#start zookeeper in terminal 0
/opt/kafka/bin/zookeeper-server-start.sh config/zookeeper.properties


#start kafka broker in terminal 1
KAFKA_HEAP_OPTS=’"-Xmx1000M -Xms1000M"’ 
KAFKA_OPTS="-javaagent:/opt/kafka/prometheus/jmx_prometheus_javaagent-0.3.0.jar=7071:/opt/kafka/prometheus/kafka-0–8–2.yml" 
JMX_PORT=7071
/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties  


#start kafka consumer in terminal 2
KAFKA_OPTS="-javaagent:/opt/kafka/prometheus/jmx_prometheus_javaagent-0.3.0.jar=7072:/opt/kafka/prometheus/kafka-0–8–2.yml" 
JMX_PORT=7072
/opt/kafka/bin/kafka-console-consumer.sh — bootstrap-server 0.0.0.0:9092 — topic test — from-beginning

#start kafka producer in terminal 3
KAFKA_OPTS="-javaagent:/opt/kafka/prometheus/jmx_prometheus_javaagent-0.3.0.jar=7073:/opt/kafka/prometheus/kafka-0–8–2.yml" 
JMX_PORT=7073
/opt/kafka/bin/kafka-console-producer.sh — broker-list 0.0.0.0:9092 — topic test

完成上述步骤后，zookeeper和kafka运行正常. 我可以在生产者终端中键入一条消息，并在消费者控制台上收到它.但是，在Prometheus上看不到kafka指标.为了进行调试，我通过

After above steps zookeeper and kafka is running fine. I can type in producer terminal a message and it is received on consumer console. However no kafka metrics is visible on Prometheus. To debug this I checked ports 7071/2/3 by

netstat -tlnp | grep 7071
netstat -tlnp | grep 7072
netstat -tlnp | grep 7073

导致空白响应；这意味着没有服务正在使用以上端口.我觉得JMX导出器未正确启用.

which resulted in blank response; this means no service is using above ports. I feel like JMX exporter is not enabled correctly.

您能帮我解决上述问题吗?

Can you help me with above issues?

推荐答案

从问题的外观看，您将变量放在它们自己的行上，而博客将它们放在同一行上...

From the looks of your question, you put the variables on their own lines, while the blog has them on the same line...

例如这是启动Kafka服务器的方法

e.g. This is how to start the Kafka server

KAFKA_HEAP_OPTS='"-Xmx1000M -Xms1000M"' KAFKA_OPTS='-javaagent:/opt/kafka/prometheus/jmx_prometheus_javaagent-0.3.0.jar=7071:/opt/kafka/prometheus/kafka-0–8–2.yml' JMX_PORT=7081 /opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties

否则，您需要导出变量，以便子流程将像您

Otherwise, you need to export the variables so the sub-process will pick them up like you did in your previous question, which seemed to be working fine for exposing the metrics

export KAFKA_HEAP_OPTS='"-Xmx1000M -Xms1000M"'
export KAFKA_OPTS='-javaagent:/opt/kafka/prometheus/jmx_prometheus_javaagent-0.3.0.jar=7071:/opt/kafka/prometheus/kafka-0–8–2.yml'
export JMX_PORT=7081
/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties

注意:链接到的博客不使用JMX_PORT，但是这些端口不能与导出器相同.

Note: The blog you linked to doesn't use JMX_PORT, but those ports cannot be the same as the exporter.

我还建议至少下载低于0.3的版本-> ://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/

I would also suggest at least downloading a version newer than 0.3 - https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/

并使用Kafka 2.0的配置- https://github.com/prometheus/jmx_exporter/blob/master/example_configs/kafka-2_0_0.yml

and using the configs for Kafka 2.0 - https://github.com/prometheus/jmx_exporter/blob/master/example_configs/kafka-2_0_0.yml

旁注:netstat -tlnp | grep 707会一次向您展示所有这些

Sidenote: netstat -tlnp | grep 707 would show you all them at once

这篇关于卡夫卡和JMX出口商的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！