kube-apiserver高CPU和请求 [英] kube-apiserver high CPU and requests

问题描述

我们在HA中拥有一个由Kops 1.7部署的Kubernetes 1.7.8集群，其中有三个主机.该集群有10个节点和大约400个Pod.

We have a Kubernetes 1.7.8 clusters deployed with Kops 1.7 in HA with three masters. The cluster has 10 nodes and around 400 pods.

集群中有堆，prometheus和ELK(为某些吊舱收集日志).

The cluster has heapster, prometheus, and ELK (collecting logs for some pods).

我们发现主服务器中的活动非常活跃，超过api服务器所用CPU的90％.

We are seeing a very high activity in the masters, over 90% of CPU used by the api-server.

检查普罗米修斯数，我们可以看到对kube-apiserver的近5000个请求是WATCH动词，其余的少于50个请求(GET，LIST，PATCH，PUT).

Checking prometheus numbers we can see that near 5000 requests to the kube-apiserver are WATCH verbs, the rest are less than 50 request (GET, LIST, PATCH, PUT).

几乎所有请求都通过客户端"Go-Http-client/2.0"报告(Go HTTP库的默认用户代理).

Almost all requests are reported with client "Go-Http-client/2.0" (the default User Agent for the Go HTTP library).

这是正常情况吗?

我们如何调试发送这些请求的Pod? (我们如何将源IP添加到kube-apiserver日志中?)

How can we debug which are the pods sending these requests? (How can we add the source IP to the kube-apiserver logs?)

[kube-apiserver.manifest] [1]

[kube-apiserver.manifest][1]

谢谢， 查尔斯

[1]: https://pastebin.com/nGxSXuZb

推荐答案

对于Kubernetes架构，这是正常现象，因为所有kubernetes集群组件都在调用api服务器来监视更改.

Regarding the Kubernetes architecture this is a normal behavior because all kubernetes cluster components are calling the api-server to watch for changes.

这就是为什么您的日志中有5000个以上WATCH条目的原因.请看一下 kubernetes集群如何由kube api服务器管理，并且主节点通信的组织方式

That is why you have more than 5000 WATCH entries in your logs. Please take a look how the kubernetes cluster is managed by kube api server and how the master-node comunication is organized

这篇关于kube-apiserver高CPU和请求的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！