芹菜任务优先级 [英] Celery Task Priority

问题描述

我想使用Celery管理任务.我希望有一个任务队列(并发1)，并且能够将具有不同优先级的任务推送到队列中，以便更高优先级的任务会抢占其他任务.

I want to manage tasks using Celery. I want to have a single task queue (with concurrency 1) and be able to push tasks onto the queue with different priorities such that higher priority tasks will preempt the others.

我正在将三个任务添加到队列中，如下所示:

I am adding three tasks to a queue like so:

add_tasks.py

from tasks import example_task

example_task.apply_async((1), priority=1)
example_task.apply_async((2), priority=3)
example_task.apply_async((3), priority=2)

我有以下配置:

tasks.py

from __future__ import absolute_import, unicode_literals
from celery import Celery
from kombu import Queue, Exchange
import time

app = Celery('tasks', backend='rpc://', broker='pyamqp://')

app.conf.task_queues = [Queue('celery', Exchange('celery'), routing_key='celery', queue_arguments={'x-max-priority': 10})]


@app.task
def example_task(task_num):
    time.sleep(3)
    print('Started {}'.format(task_num)
    return True

我希望添加的第二个任务在第三个任务之前运行，因为它具有更高的优先级，但没有.它们按照添加的顺序运行.

I expect the second task I added to run before the third, because it has higher priority but it doesn't. They run in the order added.

我正在关注文档，并认为我已经正确配置了应用程序.

I am following the docs and thought I had configured the app correctly.

我做错了还是我误解了优先功能?

Am I doing something wrong or am I misunderstanding the priority feature?

推荐答案

队列有可能没有机会对消息进行优先级排序(因为它们是在排序发生之前下载的).尝试以下两种设置(根据需要适应您的项目):

There is a possibility that the queue has no chance to prioritize the messages (because they get downloaded before the sorting happens). Try with these two settings (adapt to your project as needed):

CELERY_ACKS_LATE = True
CELERYD_PREFETCH_MULTIPLIER = 1

默认情况下，预取乘数为4.

Prefetch multiplier is 4 by default.

我已经开发了一个示例应用程序，可以在很小的范围内实现Celery的优先任务.请在此处查看.在开发它时，我遇到了一个非常相似的问题，设置的更改实际上解决了它.

I had developed a sample application to realize Celery's priority tasking on a very small scale. Please have a look at it here. While developing it, I had encountered a very similar problem and this change in settings actually solved it.

请注意，您还需要RabbitMQ版本3.5.0或更高版本.

Note that you also require RabbitMQ version 3.5.0 or higher.

这篇关于芹菜任务优先级的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！