从多个队列中消费的芹菜工人如何决定从头开始消费哪个？ [英] How does a Celery worker consuming from multiple queues decide which to consume from first?

问题描述

我正在使用Celery执行异步后台任务，并以Redis为后端。我对以下情况下芹菜工人的行为感兴趣：

I am using Celery to perform asynchronous background tasks, with Redis as the backend. I'm interested in the behaviour of a Celery worker in the following situation:

我正在使用 celeryd 。通过 -Q 选项，已为该工作人员分配了两个队列来消费：

I am running a worker as a daemon using celeryd. This worker has been assigned two queues to consume through the -Q option:

celeryd -E -Q queue1,queue2

工人如何决定从何处取货是从 queue1 或 queue2 中随机消耗一个任务吗？

How does the worker decide where to fetch the next task to consume from? Does it randomly consume a task from either queue1 or queue2? Will it prioritise fetching from queue1 because it is first in the list of arguments passed to -Q?

推荐答案

根据我的测试，它处理多个队列循环样式。

From my testing, it processes multiple queues round-robin style.

如果我使用此测试代码：

If I use this test code:

from celery import task
import time


@task
def my_task(item_id):
    time.sleep(0.5)
    print('Processing item "%s"...' % item_id)


def add_items_to_queue(queue_name, items_count):
    for i in xrange(0, items_count):
        my_task.apply_async(('%s-%d' % (queue_name, i),), queue=queue_name)


add_items_to_queue('queue1', 10)
add_items_to_queue('queue2', 10)
add_items_to_queue('queue3', 5)

然后开始队列（使用django-celery）： / p>

And start the queue with (using django-celery):

`manage.py celery worker -Q queue1,queue2,queue3`

它utputs：

Processing item "queue1-0"...
Processing item "queue3-0"...
Processing item "queue2-0"...
Processing item "queue1-1"...
Processing item "queue3-1"...
Processing item "queue2-1"...
Processing item "queue1-2"...
Processing item "queue3-2"...
Processing item "queue2-2"...
Processing item "queue1-3"...
Processing item "queue3-3"...
Processing item "queue2-3"...
Processing item "queue1-4"...
Processing item "queue3-4"...
Processing item "queue2-4"...
Processing item "queue1-5"...
Processing item "queue2-5"...
Processing item "queue1-6"...
Processing item "queue2-6"...
Processing item "queue1-7"...
Processing item "queue2-7"...
Processing item "queue1-8"...
Processing item "queue2-8"...
Processing item "queue1-9"...
Processing item "queue2-9"...

因此，它会从每个队列中提取一个项目，然后再继续执行n即使在queue2&之前已发布所有queue1任务，也可以扩展queue1项目。 3项任务。

So it pulls one item from each queue before going on to the next queue1 item even though ALL of the queue1 tasks were published before the queue2 & 3 tasks.

注意：正如@WarLord所指出的，只有在 CELERYD_PREFETCH_MULTIPLIER 设置为1。如果大于1，则表示将从批处理中的队列中提取项目。因此，如果将PREFETCH_MULTIPLIER设置为4的情况下有4个进程，那意味着马上要从队列中拉出16个项目，因此您将不会获得如上的确切输出，但仍会大致跟随循环。

Note: As @WarLord pointed out, this exact behavior will only work when CELERYD_PREFETCH_MULTIPLIER is set to 1. If it's greater than 1, then that means items will be fetched from the queue in batches. So if you have 4 processes with the PREFETCH_MULTIPLIER set to 4, that means there will be 16 items pulled from the queue right off the bat, so you won't get the exact output as above, but it will still roughly follow round-robin.

这篇关于从多个队列中消费的芹菜工人如何决定从头开始消费哪个？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！