container_memory_usage_bytes 按部署名称 [英] container_memory_usage_bytes by deployment name
问题描述
给定一个 kubernetes 集群:
至于:
<块引用>我想用指标实现的目标
kube_pod_container_resource_limits_memory_bytes
.假设部署中的每个 Pod 都具有相同的限制,您可以使用以下查询获取标记有标签的部署的内存限制,如上例所示:
kube_pod_labels{label_app=ubuntu-with-limits"} * on (pod) group_right(label_app) kube_pod_container_resource_limits_memory_bytes{namespace=memory-testing", pod=~".*"}
您可以在此查询上应用
avg()
,mean()
,max()
等函数来获取将成为您的内存限制:avg(kube_pod_labels{label_app=ubuntu-with-limits"} * on (pod) group_right(label_app) kube_pod_container_resource_limits_memory_bytes{namespace=memory-testing", pod=~".*";}) 由 (label_app)
如果您使用
VPA
,您的内存限制可能会有所不同.在这种情况下,您可以同时显示所有这些或使用avg()
来获得所有部署"的平均值.作为上述解决方案的解决方法,您可以尝试使用如下正则表达式:
container_memory_usage_bytes{pod=~"^ubuntu-.{6,10}-.{5}"}
Given a kubernetes cluster with:
I like to use the metric
container_memory_usage_bytes
but select bydeployment_name
instead ofpod
.Selectors like
container_memory_usage_bytes{pod_name=~"foo-.+"}
if thedeployment_name=foo
are great as long there is not a deployment withdeployment_name=foo-bar
.The same I'd like to achieve with the metric
kube_pod_container_resource_limits_memory_bytes
.Is there a way to achieve this?
解决方案TL;DR
There is no straightforward way to query prometheus by a
deployment-name
.You can query memory usage of a specific deployment by using deployment's labels.
Used query:
sum( kube_pod_labels{label_app=~"ubuntu.*"} * on (pod) group_right(label_app) container_memory_usage_bytes{namespace="memory-testing", container=""} ) by (label_app)
There is an awesome article which explains the concepts behind this query. I encourage you to read it:
I've included an explanation with example below.
The selector mentioned in the question:
container_memory_usage_bytes{pod_name=~"foo-.+"}
.+
- match any string but not an empty stringwith pods like:
foo-12345678-abcde
- will match (deploymentfoo
)foo-deployment-98765432-zyxzy
- will match (deploymentfoo-deployment
)
As shown above it will match for both pods and for both deployments.
For more reference:
As mentioned earlier, you can use labels from your deployment to pinpoint the resource used by your specific deployment.
Assuming that:
- There are 2 deployments in
memory-testing
namespace:ubuntu
with 3 replicasubuntu-additional
with 3 replicas
- Above deployments have labels the same as their names (they can be different):
app: ubuntu
app: ubuntu-additional
- Kubernetes cluster version
1.18.X
Why do I specify Kubernetes version?
Kubernetes 1.16 will remove the duplicate
pod_name
andcontainer_name
metric labels from cAdvisor metrics. For the 1.14 and 1.15 release allpod
,pod_name
,container
andcontainer_name
were available as a grace period.This means that you will need to substitute the parameters like:
pod
withpod_name
container
withcontainer_name
To deploy Prometheus and other tools to monitor the cluster I used: Github.com: Coreos: Kube-prometheus
The pods in
ubuntu
deployment are configured to generate artificial load (stress-ng
). This is done to show how to avoid situation where the used resources are doubled.The resources used by pods in
memory-testing
namespace:$ kubectl top pod --namespace=memory-testing NAME CPU(cores) MEMORY(bytes) ubuntu-5b5d6c56f6-cfr9g 816m 280Mi ubuntu-5b5d6c56f6-g6vh9 834m 278Mi ubuntu-5b5d6c56f6-sxldj 918m 288Mi ubuntu-additional-84bdf9b7fb-b9pxm 0m 0Mi ubuntu-additional-84bdf9b7fb-dzt72 0m 0Mi ubuntu-additional-84bdf9b7fb-k5z6w 0m 0Mi
If you were to query Prometheus with below query:
container_memory_usage_bytes{namespace="memory-testing", pod=~"ubuntu.*"}
You would get output similar to one below (it's cut to show only one pod for example purposes, by default it would show all pods with
ubuntu
in name and inmemory-testing
namespace):container_memory_usage_bytes{endpoint="https-metrics",id="/kubepods/besteffort/podb96dea39-b388-471e-a789-8c74b1670c74",instance="192.168.0.117:10250",job="kubelet",metrics_path="/metrics/cadvisor",namespace="memory-testing",node="node1",pod="ubuntu-5b5d6c56f6-cfr9g",service="kubelet"} 308559872 container_memory_usage_bytes{container="POD",endpoint="https-metrics",id="/kubepods/besteffort/podb96dea39-b388-471e-a789-8c74b1670c74/312980f90e6104d021c12c376e83fe2bfc524faa4d4cee6553182d0fa2e007a1",image="k8s.gcr.io/pause:3.2",instance="192.168.0.117:10250",job="kubelet",metrics_path="/metrics/cadvisor",name="k8s_POD_ubuntu-5b5d6c56f6-cfr9g_memory-testing_b96dea39-b388-471e-a789-8c74b1670c74_0",namespace="memory-testing",node="node1",pod="ubuntu-5b5d6c56f6-cfr9g",service="kubelet"} 782336 container_memory_usage_bytes{container="ubuntu",endpoint="https-metrics",id="/kubepods/besteffort/podb96dea39-b388-471e-a789-8c74b1670c74/1b93889a3e7415ad3fa040daf89f3f6bc77e569d85069de518267666ede8e21c",image="ubuntu@sha256:55cd38b70425947db71112eb5dddfa3aa3e3ce307754a3df2269069d2278ce47",instance="192.168.0.117:10250",job="kubelet",metrics_path="/metrics/cadvisor",name="k8s_ubuntu_ubuntu-5b5d6c56f6-cfr9g_memory-testing_b96dea39-b388-471e-a789-8c74b1670c74_0",namespace="memory-testing",node="node1",pod="ubuntu-5b5d6c56f6-cfr9g",service="kubelet"} 307777536
In this point you will need to choose which metric you will be using. In this example I used the first one. For more of a deep dive please take a look on this articles:
- Blog.freshtracks.io: A deep dive into kubernetes metrics part 3 container resource metrics
- Ianlewis.org: Almighty pause container
If we were to aggregate this metrics with
sum (QUERY) by (pod)
we would have in fact doubled our reported used resources.Dissecting the main query:
container_memory_usage_bytes{namespace="memory-testing", container=""}
Above query will output records with used memory metric for each pod. The
container=""
parameter is used to get only one record (mentioned before) which does not havecontainer
parameter.kube_pod_labels{label_app=~"ubuntu.*"}
Above query will output record with pods and it's labels with regexp of
ubuntu.*
kube_pod_labels{label_app=~"ubuntu.*"} * on (pod) group_right(label_app) container_memory_usage_bytes{namespace="memory-testing", container=""}
Above query will match the
pod
fromkube_pod_labels
withpod
ofcontainer_memory_usage_bytes
and add thelabel_app
to each of the records.sum (kube_pod_labels{label_app=~"ubuntu.*"} * on (pod) group_right(label_app) container_memory_usage_bytes{namespace="memory-testing", container=""}) by (label_app)
Above query will sum the records by
label_app
.After that you should be able to get the query that will sum the used memory by a label (and in fact a Deployment).
As for:
The same I'd like to achieve with the metric
kube_pod_container_resource_limits_memory_bytes
.You can use below query to get the memory limit for the deployment tagged with labels as in previous example, assuming that each pod in a deployment is having the same limits:
kube_pod_labels{label_app="ubuntu-with-limits"} * on (pod) group_right(label_app) kube_pod_container_resource_limits_memory_bytes{namespace="memory-testing", pod=~".*"}
You can apply functions like
avg()
,mean()
,max()
on this query to get the single number that will be your memory limit:avg(kube_pod_labels{label_app="ubuntu-with-limits"} * on (pod) group_right(label_app) kube_pod_container_resource_limits_memory_bytes{namespace="memory-testing", pod=~".*"}) by (label_app)
Your memory limits can vary if you use
VPA
. In that situation you could show all of them simultaneously or use theavg()
to get the average for all of the "deployment".
As a workaround to above solutions you could try to work with regexp like below:
container_memory_usage_bytes{pod=~"^ubuntu-.{6,10}-.{5}"}
这篇关于container_memory_usage_bytes 按部署名称的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!