当 Pods 通常需要低 CPU 但定期扩展时如何使用 K8S HPA 和自动缩放器 [英] How to use K8S HPA and autoscaler when Pods normally need low CPU but periodically scale

问题描述

我正在尝试确定与 K8S 一起使用的可靠设置，以使用 HPA 和自动缩放器扩展我的部署之一.我想尽量减少过度使用的资源量，但允许它根据需要进行扩展.

I am trying to determine a reliable setup to use with K8S to scale one of my deployments using an HPA and an autoscaler. I want to minimize the amount of resources overcommitted but allow it to scale up as needed.

我有一个管理 REST API 服务的部署.大多数情况下，该服务的使用率非常低(0m-5m cpu).但在一天或一周内，它会周期性地以 5-10 个 CPU (5000m-10000m) 的数量级使用率飙升.

I have a deployment that is managing a REST API service. Most of the time the service will have very low usage (0m-5m cpu). But periodically through the day or week it will spike to much higher usage on the order of 5-10 CPUs (5000m-10000m).

我的初始配置是:

• 部署:1 个副本

"resources": {
   "requests": {
     "cpu": 0.05
   },
   "limits": {
      "cpu": 1.0
   }
}

• HPA:

"spec": {
   "maxReplicas": 25,
   "metrics": [
      {
         "resource": {
         "name": "cpu",
         "target": {
            "averageValue": 0.75,
            "type": "AverageValue"
         }
         },
         "type": "Resource"
      }
   ],
   "minReplicas": 1,
   ...
}

这是在运行自动调节程序的 AWS EKS 集群上运行的.所有实例都有 2 个 CPU.目标是随着 CPU 使用率的上升，HPA 将分配一个不可调度的新 pod，然后自动缩放器将分配一个新节点.当我向服务添加负载时，第一个 Pod 的 CPU 使用率最高会飙升至大约 90-95%.

This is running on an AWS EKS cluster with autoscaler running. All instances have 2 CPUs. The goal is that as the CPU usage goes up the HPA will allocate a new pod that will be unschedulable and then the autoscaler will allocate a new node. As I add load on the service, the CPU usage for the first pod spikes up to approximately 90-95% at max.

我遇到了两个相关的问题:

I am running into two related problems:

1. 小请求大小

通过使用如此小的请求值(cpu:0.05)，即使在高负载的情况下，新请求的 pod 也可以轻松地调度到当前节点上.因此，自动缩放器永远不会找到无法调度且不分配新节点的 pod.我可以增加较小的请求大小并过度使用，但这意味着在没有负载的大部分时间里，我将浪费我不需要的资源.

By using such a small request value (cpu: 0.05), the newly requested pods can be easily scheduled on the current node even when it is under high load. Thus the autoscaler never find a pod that can't be scheduled and doesn't allocate a new node. I could increase the small request size and overcommit, but this then means that for the vast majority of the time when there is no load I will be wasting resources I don't need.

1. 随着分配的 Pod 越多，平均 CPU 就越低

因为所有 Pod 都分配在同一个节点上，所以一旦分配了新的 Pod，它就会开始共享该节点的可用 2 个 CPU.这反过来又减少了 pod 使用的 CPU 量，从而使平均值保持在 75% 目标以下.

Because the pods all get allocated on the same node, once a new pod is allocated it starts sharing the node's available 2 CPUs. This in turn reduces the amount of CPU used by the pod and thus keeps the average value below the 75% target.

(例如:3 个 pod，2 个 CPU ==>每个 pod 的平均 CPU 使用率最高可达 66%)

(ex: 3 pods, 2 CPUs ==> max 66% Average CPU usage per pod)

我正在这里寻求有关我应该如何思考这个问题的指导.我想我错过了一些简单的东西.

I am looking for guidance here on how I should be thinking about this problem. I think I am missing something simple.

我目前的想法是，我正在寻找一种方法，让 Pod 资源请求值在较重的负载下增加，然后在系统不需要时减少回落.这会让我倾向于使用 VPA 之类的东西，但我读过的所有内容都表明，同时使用 HPA 和 VPA 会导致非常糟糕的事情.

My current thought is that what I am looking for is a way for the Pod resource request value to increase under heavier load and then decrease back down when the system doesn't need it. That would point me toward using something like a VPA, but everything I have read says that using HPA and VPA at the same time leads to very bad things.

我认为将请求从 0.05 增加到 0.20 之类的值可能会让我处理放大的情况.但这反过来又会浪费大量资源，并且如果调度程序在现有 pod 上找到空间，则可能会遇到问题.我的示例是关于一项服务，但在生产部署中还有更多服务.我不想让节点空着，但没有使用已提交的资源.

I think increasing the request from 0.05 to something like 0.20 would probably let me handle the case of scaling up. But this will in turn waste a lot of resources and could suffer issues if the scheduler find space on an existing pod. My example is about one service but there are many more services in the production deployment. I don't want to have nodes sitting empty with committed resources but no usage.

这里最好的前进道路是什么?

What is the best path forward here?

推荐答案

听起来您需要一个将实际 CPU 利用率考虑在内的调度程序.尚不支持此功能.

Sounds like you need a Scheduler that take actual CPU utilization into account. This is not supported yet.

似乎有一个这个功能的工作:KEP - Trimaran: Real Load Aware Scheduling 使用 TargetLoadPackin 插件.另请参阅实际负载平均和可用内存的新调度程序优先级.

There seem to be work on a this feature: KEP - Trimaran: Real Load Aware Scheduling using TargetLoadPackin plugin. Also see New scheduler priority for real load average and free memory.

同时，如果 CPU 限制为 1 核，并且节点在高 CPU 使用率下自动缩放，听起来它应该可以工作如果节点远大于 Pod 的 CPU 限制.例如.尝试使用具有 4 个或更多内核的节点，并且可能为 Pod 提供稍大的CPU 请求?

In the meanwhile, if the CPU limit is 1 Core, and the Nodes autoscale under high CPU utilization, it sounds like it should work if the nodes is substantially bigger than the CPU limits for the pods. E.g. try with nodes that has 4 Cores or more and possibly slightly larger CPU request for the Pod?

这篇关于当 Pods 通常需要低 CPU 但定期扩展时如何使用 K8S HPA 和自动缩放器的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！