群集自动缩放器未缩减 [英] Cluster autoscaler not downscaling
问题描述
我在 google kubernetes引擎(GKE)中设置了一个区域集群.节点组是每个区域中的单个 vm(共3个).我的部署受HPA控制，最少需要 3个副本. 节点组被配置为自动缩放(集群自动缩放，也称为CA). 问题场景:
I have a regional cluster set up in google kubernetes engine (GKE). The node group is a single vm in each region (3 total). I have a deployment with 3 replicas minimum controlled by a HPA. The nodegroup is configured to be autoscaling (cluster autoscaling aka CA). The problem scenario:
更新部署映像. Kubernetes自动创建新的Pod，并且CA标识需要一个新的节点.我现在有4. 当所有新的Pod已启动时，旧的Pod将被删除，这意味着我与前一分钟的CPU请求完全相同.但是在最大缩减时间10分钟之后，我仍然有4个节点.
Update deployment image. Kubernetes automatically creates new pods and the CA identifies that a new node is needed. I now have 4. The old pods get removed when all new pods have started, which means I have the exact same CPU request as the minute before. But the after the 10 min maximum downscale time I still have 4 nodes.
对节点的CPU请求现在为:
The CPU requests for the nodes is now:
CPU Requests CPU Limits Memory Requests Memory Limits
------------ ---------- --------------- -------------
358m (38%) 138m (14%) 516896Ki (19%) 609056Ki (22%)
--
CPU Requests CPU Limits Memory Requests Memory Limits
------------ ---------- --------------- -------------
800m (85%) 0 (0%) 200Mi (7%) 300Mi (11%)
--
CPU Requests CPU Limits Memory Requests Memory Limits
------------ ---------- --------------- -------------
510m (54%) 100m (10%) 410Mi (15%) 770Mi (29%)
--
CPU Requests CPU Limits Memory Requests Memory Limits
------------ ---------- --------------- -------------
823m (87%) 158m (16%) 484Mi (18%) 894Mi (33%)
38％的节点正在运行:
The 38% node is running:
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits
--------- ---- ------------ ---------- --------------- -------------
kube-system event-exporter-v0.1.9-5c8fb98cdb-8v48h 0 (0%) 0 (0%) 0 (0%) 0 (0%)
kube-system fluentd-gcp-v2.0.17-q29t2 100m (10%) 0 (0%) 200Mi (7%) 300Mi (11%)
kube-system heapster-v1.5.2-585f569d7f-886xx 138m (14%) 138m (14%) 301856Ki (11%) 301856Ki (11%)
kube-system kube-dns-autoscaler-69c5cbdcdd-rk7sd 20m (2%) 0 (0%) 10Mi (0%) 0 (0%)
kube-system kube-proxy-gke-production-cluster-default-pool-0fd62aac-7kls 100m (10%) 0 (0%) 0 (0%) 0 (0%)
我怀疑它不会降级，因为heapster或kube-dns-autoscaler. 但是85％的豆荚包含:
I suspect it wont downscale because heapster or kube-dns-autoscaler. But the 85% pod contains:
Namespace Name CPU Requests CPU Limits Memory Requests Memory Limits
--------- ---- ------------ ---------- --------------- -------------
kube-system fluentd-gcp-v2.0.17-s25bk 100m (10%) 0 (0%) 200Mi (7%) 300Mi (11%)
kube-system kube-proxy-gke-production-cluster-default-pool-7ffeacff-mh6p 100m (10%) 0 (0%) 0 (0%) 0 (0%)
my-deploy my-deploy-54fc6b67cf-7nklb 300m (31%) 0 (0%) 0 (0%) 0 (0%)
my-deploy my-deploy-54fc6b67cf-zl7mr 300m (31%) 0 (0%) 0 (0%) 0 (0%)
流利的和kube-proxy的pod出现在每个节点上，因此我假设没有该节点就不需要它们.这意味着我的部署可以重定位到其他节点，因为它只有300m的请求(31％，因为只有94％的节点CPU是可分配的).
The fluentd and kube-proxy pods are present on every node, so I assume they are not needed without the node. Which means that my deployment could be relocated to the other nodes since it only has a request of 300m (31% since only 94% of node CPU is allocatable).
所以我认为我会检查日志.但是，如果我运行kubectl get pods --all-namespaces，则CA的GKE上没有可见的pod.如果我使用命令kubectl get configmap cluster-autoscaler-status -n kube-system -o yaml，它只会告诉我是否要扩展，而不是为什么或为什么不.
另一种选择是查看主节点中的/var/log/cluster-autoscaler.log.我在所有4个节点上进行SSH:ed操作，仅找到一个gcp-cluster-autoscaler.log.pos文件，该文件显示:/var/log/cluster-autoscaler.log 0000000000000000 0000000000000000意味着该文件应该在此处但为空.
根据 FAQ 的最后一个选择是检查吊舱的事件，但据我所知它们是空的.
So I figured that Ill check the logs. But if I run kubectl get pods --all-namespaces there are no pod visible on GKE for the CA. And if I use the command kubectl get configmap cluster-autoscaler-status -n kube-system -o yaml it only tells me if it is about to scale, not why or why not.
Another option is to look at /var/log/cluster-autoscaler.log in the master node. I SSH:ed in the all 4 nodes and only found a gcp-cluster-autoscaler.log.pos file that says: /var/log/cluster-autoscaler.log 0000000000000000 0000000000000000 meaning the file should be right there but is empty.
Last option according to the FAQ, is to check the events for the pods, but as far as i can tell they are empty.
任何人都知道为什么它不会缩小规模或至少在哪里找到日志?
Anyone know why it wont downscale or atleast where to find the logs?
推荐答案
帮助自己提高知名度.
问题在于，除非常见问题解答是同时满足的. 因此，可以说我有100个具有51％CPU请求的节点.它仍然不会考虑缩小规模.
The problem is that the CA never considers moving anything unless all the requirements mentioned in the FAQ are met at the same time. So lets say I have 100 nodes with 51% CPU requests. It still wont consider downscaling.
一种解决方案是将CA检查的价值提高到现在的50％.但是很遗憾，GKE不支持，请参见Google支持@GalloCedrone的答案:
One solution is to increase the value at which CA checks, now 50%. But unfortunately that is not supported by GKE, see answer from google support @GalloCedrone:
此外，我知道这个值听起来可能太低了，有人可能有兴趣将其保持在85％或90％的水平，以避免出现您的情况. 当前有一个开放的功能请求，使用户可以修改标志"--scale-down-utilization-threshold"，但尚未实现.
Moreover I know that this value might sound too low and someone could be interested to keep as well a 85% or 90% to avoid your scenario. Currently there is a feature request open to give the user the possibility to modify the flag "--scale-down-utilization-threshold", but it is not implemented yet.
我发现的解决方法是减少Pod的CPU请求(从100m而不是300m)，并使Horizontal Pod Autoscaler(HPA)按需创建更多.这对我来说很好，但是如果您的应用程序不适合许多小型实例，那么您就不走运了.如果总利用率低，也许是一项cron作业会封锁一个节点?
The workaround I found is to decrease the CPU request (100m instead of 300m) of the pods and have the Horizontal Pod Autoscaler (HPA) create more on demand. This is fine for me but if your application is not suitable for many small instances you are out of luck. Perhaps a cron job that cordons a node if the total utilization is low?
这篇关于群集自动缩放器未缩减的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
