有没有办法仅在 Kubernetes 中使用 HorizontalPodAutoscaler 处理消息(pod 完成其任务)时缩小 pod 的规模? [英] Is there a way to downscale pods only when message is processed (the pod finished its task) with the HorizontalPodAutoscaler in Kubernetes?
问题描述
我使用 prometheus 适配器设置了具有自定义指标的 Kubernetes Horizontal Pod Autoscaler https://github.com/DirectXMan12/k8s-prometheus-adapter.Prometheus 正在监控 rabbitmq，我正在关注 rabbitmq_queue_messages 指标.来自队列的消息由 Pod 拾取，然后进行一些处理，这可能会持续几个小时.
I'v setup Kubernetes Horizontal Pod Autoscaler with custom metrics using the prometheus adapter https://github.com/DirectXMan12/k8s-prometheus-adapter. Prometheus is monitoring rabbitmq, and Im watching the rabbitmq_queue_messages metric. The messages from the queue are picked up by the pods, that then do some processing, which can last for several hours.
放大和缩小是根据队列中的消息数量进行的.
The scale-up and scale-down is working based on the number of messages in the queue.
问题:当 pod 完成处理并确认消息时，这将降低 num.队列中的消息，这将触发 Autoscaler 终止 pod.如果我有多个 pod 进行处理并且其中一个完成，如果我没有记错的话，Kubernetes 可以终止仍在处理自己的消息的 pod.这是不可取的，因为 pod 正在执行的所有处理都将丢失.
The problem: When a pod finishes the processing and acks the message, that will lower the num. of messages in the queue, and that would trigger the Autoscaler terminate a pod. If I have multipe pods doing the processing and one of them finishes, if Im not mistaking, Kubernetes could terminate a pod that is still doing the processing of its own message. This wouldnt be desirable as all the processing that the pod is doing would be lost.
有没有办法克服这个问题，或者有其他方法可以解决这个问题吗?
Is there a way to overcome this, or another way how this could be acheveed?
这里是 Autoscaler 配置:
here is the Autoscaler configuration:
kind: HorizontalPodAutoscaler
apiVersion: autoscaling/v2beta1
metadata:
name: sample-app-rabbitmq
namespace: monitoring
spec:
scaleTargetRef:
# you created above
apiVersion: apps/v1
kind: Deployment
name: sample-app
minReplicas: 1
maxReplicas: 10
metrics:
- type: Object
object:
target:
kind: Service
name: rabbitmq-cluster
metricName: rabbitmq_queue_messages_ready
targetValue: 5
推荐答案
Horizontal Pod Autoscaler 不是为长时间运行的任务而设计的，因此不太适合.如果您需要为每条消息生成一个长时间运行的处理任务，我会采用以下两种方法之一:
Horizontal Pod Autoscaler is not designed for long-running tasks, and will not be a good fit. If you need to spawn one long-running processing tasks per message, I'd take one of these two approaches:
- 使用任务队列，例如 Celery.它旨在解决您的确切问题:有一个需要分发给工作人员的任务队列，并确保任务运行完成.Kubernetes 甚至提供了此设置的官方示例.
- 如果不想引入其他组件，例如 Celery，可以生成一个 Kubernetes job 用于您自己的每条传入消息.Kubernetes 将确保作业至少运行到完成一次 - 如果 pod 死亡，则重新安排 pod 等.在这种情况下，您需要编写一个脚本来读取 RabbitMQ 消息并自行为它们创建作业.
- Use a task queue such as Celery. It is designed to solve your exact problem: have a queue of tasks that needs to be distributed to workers, and ensure that the tasks run to completion. Kubernetes even provides an official example of this setup.
- If you don't want to introduce another component such as Celery, you can spawn a Kubernetes job for every incoming message by yourself. Kubernetes will make sure that the job runs to completion at least once - reschedule the pod if it dies, etc. In this case you will need to write a script that reads RabbitMQ messages and creates jobs for them by yourself.
在这两种情况下，请确保您还拥有集群自动缩放器 启用，以便在当前节点不足以处理负载时自动配置新节点.
In both cases, make sure you also have Cluster Autoscaler enabled so that new nodes get automatically provisioned if your current nodes are not sufficient to handle the load.
这篇关于有没有办法仅在 Kubernetes 中使用 HorizontalPodAutoscaler 处理消息(pod 完成其任务)时缩小 pod 的规模?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
