如何使用Kubernetes进行扩展工作队列 [英] How to work a job queue with kubernetes with scaling

问题描述

我需要基于docker/python worker的可扩展队列处理.我的想法转向了kubernetes.但是，我不确定最好的控制器/服务.

基于azure函数，我收到了传入的http流量，将简单消息添加到了存储队列.这些消息需要处理，并将结果反馈到结果队列中.

为了处理那些队列消息，我开发了python代码来循环队列并处理这些作业.每次成功循环之后，该消息将从源队列中删除，并将结果写入结果队列.队列为空后，代码就存在了.

因此，我创建了一个运行python代码的docker映像.如果启动了多个容器，队列显然会更快地工作. 我还实现了新的Azure Kubernetes服务以进行扩展. 当我是kubernetes的新手时，我读到了在队列准备工作之前工作队列的工作范式.我简单的yaml模板如下所示:

apiVersion: batch/v1
kind: Job
metadata:
  name: myjob
spec:
  parallelism: 4
  template:
    metadata:
      name: myjob
    spec:
      containers:
      - name: c
        image: repo/image:tag

我现在的问题是，该作业无法重新启动.

通常，队列中充满了一些条目，然后有一段时间没有任何反应.然后，更大的队列又会到达，需要尽快处理.当然，我想再次运行该作业，但这似乎是不可能的.另外，如果队列中什么也没有，我想将占用空间减小到最小.

所以我的问题是，在这种情况下我应该使用哪种体系结构，是否有简单的yaml示例?

解决方案

这可能是一个愚蠢/hacky"的答案，但是它简单，健壮，而且我已经在生产系统中使用了几个月. /p>

我有一个类似的系统，其中有一个队列，有时会被清空，有时会被猛烈撞击.我写了类似的队列处理器，它一次处理队列中的一条消息，如果队列为空，则终止.它设置为在Kubernetes作业中运行.

诀窍是这样的:我创建了一个CronJob来定期启动该作业的一个新实例，并且该作业允许无限的并行性.如果队列为空，它将立即终止(缩小").如果队列被猛击并且上一个作业尚未完成，则另一个实例启动(向上扩展").

无需费心查询队列和扩展状态集或其他任何内容，并且如果队列为空，则不会消耗任何资源.您可能需要调整CronJob间隔，以微调它对填充队列的反应速度，但它应该反应良好.

I need a scalable queue handling based on docker/python worker. My thought went towards kubernetes. However, I am unsure about the best controller/service.

Based on azure functions I get incoming http traffic adding simple messages to a storage queue. Those messages need to be worked on and the results fed back into a result queue.

To process those queue messages I developed python code looping the queue and working on those jobs. After each successful loop, the message will be removed from the source-queue and the result written into the result-queue. Once the queue is empty the code exists.

So I created a docker image that runs the python code. If more than one container is started the queue gets worked faster obviously. I also implemented the new Azure Kubernetes Services to scale that. While being new to kubernetes I read about the job paradigm to work a queue until the job is ready. My simple yaml template looks like this:

apiVersion: batch/v1
kind: Job
metadata:
  name: myjob
spec:
  parallelism: 4
  template:
    metadata:
      name: myjob
    spec:
      containers:
      - name: c
        image: repo/image:tag

My problem now is, that the job cannot be restarted.

Usually, the queue gets filled with some entries and then for a while nothing happens. Then again bigger queues can arrive that need to be worked on as fast as possible. Of course, I want to run the job again then, but that seems not possible. Also, I want to reduce the footprint to a minimum if nothing is in the queue.

So my question is, what architecture/constructs should I use for this scenario and are there simple yaml examples for that?

解决方案

This may be a "goofy/hacky" answer, but it's simple, robust, and I've been using it in a production system for months now.

I have a similar system where I have a queue that sometimes is emptied out and sometimes gets slammed. I wrote my queue processor similarly, it handles one message in the queue at a time and terminates if the queue is empty. It is set up to run in a Kubernetes job.

The trick is this: I created a CronJob to regularly start one single new instance of the job, and the job allows infinite parallelism. If the queue is empty, it immediately terminates ("scales down"). If the queue is slammed and the last job hadn't finished yet, another instance starts ("scales up").

No need to futz with querying the queue and scaling a statefulset or anything, and no resources are consumed if the queue is sitting empty. You may have to adjust the CronJob interval to fine tune how fast it reacts to the queue filling up, but it should react pretty well.

这篇关于如何使用Kubernetes进行扩展工作队列的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！