根据每个Pod的活动连接数扩展GKE Pod [英] Scaling GKE pods based on number of active connections per pod
问题描述
我有一个正在运行的GKE集群，并且使用目标CPU利用率指标来进行HPA.可以，但是CPU利用率不是我们的最佳扩展指标.分析表明，活动连接数是总体平台负载的良好指标，因此，我们希望将此作为主要的扩展指标.
I have a running GKE cluster with an HPA using a target CPU utilisation metric. This is OK but CPU utilisation is not the best scaling metric for us. Analysis suggests that active connection count is a good indicator of general platform load and thus, we'd like to look into this as our primary scaling metric.
为此，我为我们使用的NGINX入口启用了自定义指标.从这里我们可以看到活动的连接数，请求速率等.
To this end I have enabled custom metrics for the NGINX ingress that we use. From here we can see active connection counts, request rates, etc.
以下是使用NGINX自定义指标的HPA规范:
Here is the HPA specification using the NGINX custom metric:
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: hpa-uat-active-connections
namespace: default
spec:
minReplicas: 3
maxReplicas: 6
metrics:
- type: Pods
pods:
metricName: custom.googleapis.com|nginx-ingress-controller|nginx_ingress_controller_nginx_process_connections
selector:
matchLabels:
metric.labels.state: active
resource.labels.cluster_name: "[redacted]"
targetAverageValue: 5
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: "[redacted]"
但是，尽管此规范的确部署正常，但我总是从HPA获得以下输出:
However, while this specification does deploy OK, I always get this output from the HPA:
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
hpa-uat-active-connections Deployment/[redacted] <unknown>/5 3 6 3 31s
简而言之，目标值是未知"，到目前为止，我仍然无法理解/解决原因.确实存在自定义指标:
In short, the target value is "unknown" and I have so far failed to understand / resolve why. The custom metric is indeed present:
kubectl get --raw"/apis/external.metrics.k8s.io/v1beta1/namespaces/default/custom.googleapis.com|nginx-ingress-controller|nginx_ingress_controller_nginx_process_connections?labelSelector=metric.labels.state%3Dactive， resource.labels.cluster_name％3D [acted]"| jq
kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/default/custom.googleapis.com|nginx-ingress-controller|nginx_ingress_controller_nginx_process_connections?labelSelector=metric.labels.state%3Dactive,resource.labels.cluster_name%3D[redacted]" | jq
哪个给:
{
"kind": "ExternalMetricValueList",
"apiVersion": "external.metrics.k8s.io/v1beta1",
"metadata": {
"selfLink": "/apis/external.metrics.k8s.io/v1beta1/namespaces/default/custom.googleapis.com%7Cnginx-ingress-controller%7Cnginx_ingress_controller_nginx_process_connections"
},
"items": [
{
"metricName": "custom.googleapis.com|nginx-ingress-controller|nginx_ingress_controller_nginx_process_connections",
"metricLabels": {
"metric.labels.controller_class": "nginx",
"metric.labels.controller_namespace": "ingress-nginx",
"metric.labels.controller_pod": "nginx-ingress-controller-54f84b8dff-sml6l",
"metric.labels.state": "active",
"resource.labels.cluster_name": "[redacted]",
"resource.labels.container_name": "",
"resource.labels.instance_id": "[redacted]-eac4b327-stqn",
"resource.labels.namespace_id": "ingress-nginx",
"resource.labels.pod_id": "nginx-ingress-controller-54f84b8dff-sml6l",
"resource.labels.project_id": "[redacted],
"resource.labels.zone": "[redacted]",
"resource.type": "gke_container"
},
"timestamp": "2019-12-30T14:11:01Z",
"value": "1"
}
]
}
所以我确实有两个问题:
So I have two questions, really:
- (主要问题):我在这里做错了什么导致HPA无法读取该指标吗?
- 这是尝试扩展到多个Pod上的平均活动连接负载的正确方法吗?
非常感谢， 本
编辑1
kubectl全部获得
kubectl get all
NAME READY STATUS RESTARTS AGE
pod/[redacted]-deployment-7f5fbc9ddf-l9tqk 1/1 Running 0 34h
pod/[redacted]-uat-deployment-7f5fbc9ddf-pbcns 1/1 Running 0 34h
pod/[redacted]-uat-deployment-7f5fbc9ddf-tjfrm 1/1 Running 0 34h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/[redacted]-webapp-service NodePort [redacted] <none> [redacted] 57d
service/kubernetes ClusterIP [redacted] <none> [redacted] 57d
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/[redacted]-uat-deployment 3/3 3 3 57d
NAME DESIRED CURRENT READY AGE
replicaset.apps/[redacted]-uat-deployment-54b6bd5f9c 0 0 0 12d
replicaset.apps/[redacted]-uat-deployment-574c778cc9 0 0 0 35h
replicaset.apps/[redacted]-uat-deployment-66546bf76b 0 0 0 11d
replicaset.apps/[redacted]-uat-deployment-698dfbb6c4 0 0 0 4d
replicaset.apps/[redacted]-uat-deployment-69b5c79d54 0 0 0 6d17h
replicaset.apps/[redacted]-uat-deployment-6f67ff6599 0 0 0 10d
replicaset.apps/[redacted]-uat-deployment-777bfdbb9d 0 0 0 3d23h
replicaset.apps/[redacted]-uat-deployment-7f5fbc9ddf 3 3 3 34h
replicaset.apps/[redacted]-uat-deployment-9585454ff 0 0 0 6d21h
replicaset.apps/[redacted]-uat-deployment-97cbcfc6 0 0 0 17d
replicaset.apps/[redacted]-uat-deployment-c776f648d 0 0 0 10d
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
horizontalpodautoscaler.autoscaling/[redacted]-uat-deployment Deployment/[redacted]-uat-deployment 4%/80% 3 6 3 9h
推荐答案
Ok I managed to figure this out by looking up the schema for the HPA (https://docs.okd.io/latest/rest_api/apis-autoscaling/v2beta1.HorizontalPodAutoscaler.html).
简而言之，我使用了错误的指标类型(如上所示，您可以看到我使用的是"Pods"，但我应该使用外部").
In short, I was using the wrong metric type (as above you can see I am using "Pods", but I should be using "External").
正确的HPA规范是:
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: hpa-uat-active-connections
namespace: default
spec:
minReplicas: 3
maxReplicas: 6
metrics:
- type: External
external:
metricName: custom.googleapis.com|nginx-ingress-controller|nginx_ingress_controller_nginx_process_connections
metricSelector:
matchLabels:
metric.labels.state: active
resource.labels.cluster_name: [redacted]
targetAverageValue: 5
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: [redacted]
我这样做后，一切立即生效:
As soon as I did this, things worked right away:
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
hpa-uat-active-connections Deployment/bustle-webapp-uat-deployment 334m/5 (avg) 3 6 3 30s
这篇关于根据每个Pod的活动连接数扩展GKE Pod的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
