在芹菜的多个队列中确定队列的优先级? [英] Prioritizing queues among multiple queues in celery?

问题描述

我们将celery用于异步后台任务，并且有2个队列用于不同优先级的任务.我们有2个节点集群，分别为它们服务.一切都按预期进行.

We are using celery for our asynchronous background tasks and we have 2 queues for different priority tasks. We have 2 cluster of nodes serving them separately. Things are working well as expected.

问题:

我们主要收到低优先级的任务.为了优化资源利用，我想知道是否有一种方法可以配置工作程序(侦听高优先级队列)以侦听两个队列.但是，只要有某些作业存在，就从较高优先级的队列中取出作业?并回退到低优先级队列.

We get mostly low priority tasks. For optimized resource utilization, I am wondering is there a way to configure workers(listening to high priority queue) to listen to both queues. But take jobs from the higher priority queue as long as some job is there? and fallback to low priority queue otherwise.

我已经完成了@ Celery任务优先级中讨论的基于优先级的任务调度.

I have gone through the priority based task scheduling discussed @ Celery Task Priority.

但是我的问题是，不仅要对队列中的任务进行优先级排序，而且要对队列进行优先级排序.

But my questions is prioritize queues not just tasks within a queue.

推荐答案

您可以在启动工作程序时为其定义多个队列，从而部分实现此目标.

You can partially achieve this by defining multiple queues for the worker, when starting it.

您可以使用以下命令来执行此操作:另外，请在此处参考以获得更多详细信息.

You can do it with the following command: Also, refer here for more details.

celery -A proj worker -l info -Q Q1,Q2

尽管这种方法有问题.它不能通过后备方法来做到这一点.因此，收听多个队列的工作人员将资源平均分配给他们.

Though this approach has a problem. It doesn't do it with fallback kind of approach. Since, workers listening to multiple queue evenly distribute the resources among them.

因此，即使在普通优先级队列"中有某些内容时，也无法仅从高优先级队列"进行处理.可以通过为高优先级队列"分配更多的工作线程(可能为75％)和为普通优先级队列"分配25％的工作最小化.或根据您的工作量分配不同的份额.

Hence, your requirement of processing only from 'high priority queue' even when there is something in 'normal priority queue' cannot be achieved. This can be minimized by allocating more Workers (may be 75%) for 'high priority queue' and 25% for 'normal priority queue'. or different share based on you work load.

这篇关于在芹菜的多个队列中确定队列的优先级?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！