Grafana/prometheus 中没有 kafka 指标 [英] No kafka metrics in Grafana/prometheus

问题描述

我成功部署了 helm chart

我发现有人报告了类似的问题(https://groups.google.com/forum/#!searchin/prometheus-users/jvm_%7Csort:date/prometheus-users/OtYM7qGMbvA/dZ4vIfWLAgAJ)，所以我部署了旧的jmx 导出器的版本从 0.6 到 0.9，仍然没有 jvm_ 指标.

有什么我遗漏的吗?

环境:

kuberentes:AWS EKS(kubernetes 版本为 1.10.x)

公共 grafana 仪表板:kafka 概览

解决方案

刚刚实现了README中提到的jmx-exporter的拥有者:

<块引用>

此导出器旨在作为 Java 代理运行，公开 HTTP 服务器并提供本地 JVM 的指标.它也可以作为独立的 HTTP 服务器运行并抓取远程 JMX 目标，但这有各种缺点，例如更难配置和无法公开进程指标(例如，内存和 CPU 使用率).因此，强烈建议将导出器作为 Java 代理运行.

直到我看到这条评论才真正明白这是什么意思:

https://github.com/prometheus/jmx_exporter/issues/111#issuecomment-341983150

<块引用>

@brian-brazil 您能否在自述文件中添加一些提示，即 jvm_* 指标仅在使用 Java 代理时才公开?在仅使用 HTTP 服务器版本之后，我花了一两个小时进行故障排除和搜索旧问题来解决这个问题.谢谢！

因此 jmx-exporter 必须与 java agent 一起运行才能获得 jvm_ 指标.jmx_prometheus_httpserver 不支持，但是kafka helm chart默认设置.

https://github.com/kubernetes/charts/blob/master/incubator/kafka/templates/statefulset.yaml#L82

命令:- sh--exc- |陷阱退出0"术语；\尽管 :;做 \爪哇\-XX:+UnlockExperimentalVMOptions \-XX:+UseCGroupMemoryLimitForHeap \-XX:MaxRAMFraction=1 \-XshowSettings:vm \-jar \jmx_prometheus_httpserver.jar \ # <<<这里{{ .Values.prometheus.jmx.port |引用 }} \/etc/jmx-kafka/jmx-kafka-prometheus.yml &\等等$！||睡3;\完毕

I successfully deployed helm chart prometheus operator, kube-prometheus and kafka (tried both image danielqsj/kafka_exporter v1.0.1 and v1.2.0).

Install with default value mostly, rbac are enabled.

I can see 3 up nodes in Kafka target list in prometheus, but when go in Grafana, I can's see any kafka metric with kafka overview

Anything I missed or what I can check to fix this issue?

I can see metrics start with java_, kafka_, but no jvm_ and only few jmx_ metrics.

I found someone reported similar issue (https://groups.google.com/forum/#!searchin/prometheus-users/jvm_%7Csort:date/prometheus-users/OtYM7qGMbvA/dZ4vIfWLAgAJ), So I deployed with old version of jmx exporter from 0.6 to 0.9, still no jvm_ metrics.

Are there anything I missed?

env:

kuberentes: AWS EKS (kubernetes version is 1.10.x)

public grafana dashboard: kafka overview

解决方案

Just realised the owner of jmx-exporter mentioned in README:

This exporter is intended to be run as a Java Agent, exposing a HTTP server and serving metrics of the local JVM. It can be also run as an independent HTTP server and scrape remote JMX targets, but this has various disadvantages, such as being harder to configure and being unable to expose process metrics (e.g., memory and CPU usage). Running the exporter as a Java Agent is thus strongly encouraged.

Not really understood what's that meaning, until I saw this comment:

https://github.com/prometheus/jmx_exporter/issues/111#issuecomment-341983150

@brian-brazil can you add some sort of tip to the readme that jvm_* metrics are only exposed when using the Java agent? It took me an hour or two of troubleshooting and searching old issues to figure this out, after playing only with the HTTP server version. Thanks!

So jmx-exporter has to be run with java agent to get jvm_ metric. jmx_prometheus_httpserver doesn't support, but it is the default setting in kafka helm chart.

https://github.com/kubernetes/charts/blob/master/incubator/kafka/templates/statefulset.yaml#L82

command:
- sh
- -exc
- |
  trap "exit 0" TERM; \
  while :; do \
  java \
  -XX:+UnlockExperimentalVMOptions \
  -XX:+UseCGroupMemoryLimitForHeap \
  -XX:MaxRAMFraction=1 \
  -XshowSettings:vm \
  -jar \
  jmx_prometheus_httpserver.jar \              # <<< here
  {{ .Values.prometheus.jmx.port | quote }} \
  /etc/jmx-kafka/jmx-kafka-prometheus.yml & \
  wait $! || sleep 3; \
  done

这篇关于Grafana/prometheus 中没有 kafka 指标的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！