Kubernetes AWS Cloudwatch适配器未获取EKS HPA自动缩放的自定义指标值 [英] Kubernetes AWS Cloudwatch adapter not fetching custom metric value for EKS HPA autoscaling
问题描述
我正在尝试通过Kubernetes Cloudwatch适配器基于自定义Cloudwatch指标启用AWS EKS自动缩放.我已将自定义指标推送到AWS Cloudwatch,并验证它们是否出现在Cloudwatch控制台中以及是否可以使用boto3客户端get_metric_data检索到.这是我用来将自定义指标发布到Cloudwatch的代码:
I'm trying to enable AWS EKS autoscaling based on a custom Cloudwatch metric via the Kubernetes Cloudwatch adapter. I have pushed custom metrics to AWS Cloudwatch, and validated they appear in Cloudwatch console as well as are retrievable using the boto3 client get_metric_data. This is the code I use to publish my custom metric to Cloudwatch:
import boto3
from datetime import datetime
client = boto3.client('cloudwatch')
cloudwatch_response = client.put_metric_data(
Namespace='TestMetricNS',
MetricData=[
{
'MetricName': 'TotalUnprocessed',
'Timestamp': datetime.now(),
'Value': 40,
'Unit': 'Megabytes',
}
]
)
我具有以下yaml文件,用于在kubernetes中建立外部指标和hpa自动缩放器:
I have the following yaml files for establishing the external metric and the hpa autoscaler in kubernetes:
extMetricCustom.yaml:
extMetricCustom.yaml:
apiVersion: metrics.aws/v1alpha1
kind: ExternalMetric
metadata:
name: test-custom-metric
spec:
name: test-custom-metric
resource:
resource: "deployment"
queries:
- id: sqs_test
metricStat:
metric:
namespace: "TestMetricNS"
metricName: "TotalUnprocessed"
period: 60
stat: Average
unit: Megabytes
returnData: true
hpaCustomMetric.yaml
hpaCustomMetric.yaml
kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v2beta1
metadata:
name: test-scaler
spec:
scaleTargetRef:
apiVersion: apps/v1beta1
kind: Deployment
name: sqs-consumer
minReplicas: 1
maxReplicas: 4
metrics:
- type: External
external:
metricName: test-custom-metric
targetAverageValue: 2
当我评估Kubernetes Cloudwatch适配器是否正确获取我的自定义指标(kubectl get hpa)时,它始终显示该指标为0:
When I assess whether the Kubernetes Cloudwatch adapter is properly grabbing my custom metric (kubectl get hpa), it always displays that the metric is 0:
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
test-scaler Deployment/sqs-consumer 0/2 (avg) 1 4 1 161m
如何根据Cloudwatch自定义指标正确地自动缩放?
How can I properly autoscale based off my Cloudwatch custom metric?
推荐答案
在带外与OP一起工作,并且当天晚些时候仍然打开该问题的选项卡,因此在此处发布结果以供所有人参考偶然发现了它.
Worked with OP on this out-of-band and still had the tab open for this question later in the day, so posting the outcome here for posterity for anyone that stumbles upon it.
此问题的根本原因是时区冲突.指标监控器基于当前"指标.指标,但指标生成器脚本的以下代码行在未指定时区的情况下生成时间戳,并且该时间戳也位于本地时区.
The root cause of the issue was a timezone conflict. The metrics monitor was based on "current" metrics, but the the following line from the metric generator script was producing time stamps without a timezone specified and also was in a local timezone.
'Timestamp': datetime.now(),
因为没有数据",对于当前时区(由于-X UTC偏移,仅过去X个小时的数据),系统未启动缩放,因为有效值为"0"/nil/null.相反,可以指定UTC时间字符串以确保生成的度量标准是及时的:
Since there was "no data" for the current timezone (only data X hours in the past due to a -X UTC offset), the system did not initiate scaling because there was a value of "0"/nil/null effectively. Instead, a UTC time string can be specified to ensure the generated metrics are timely:
'Timestamp': datetime.utcnow(),
次要考虑的是Kubernetes节点需要访问权才能从CloudWatch轮询指标.这是通过附加此政策节点的IAM角色:
A secondary consideration was that the Kubernetes Nodes need access to poll the metrics from CloudWatch. This is done by attaching this policy to the nodes's IAM role:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"cloudwatch:GetMetricData"
],
"Resource": "*"
}
]
}
这篇关于Kubernetes AWS Cloudwatch适配器未获取EKS HPA自动缩放的自定义指标值的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!