为什么 celery 会向 rabbitmq 添加数千个队列，这些队列在任务完成后似乎仍然存在很长时间? [英] Why does celery add thousands of queues to rabbitmq that seem to persist long after the tasks completel?

问题描述

我正在使用带有 rabbitmq 后端的 celery.它在 rabbitmq 中生成了数千个队列，其中包含 0 或 1 个项目，如下所示:

I am using celery with a rabbitmq backend. It is producing thousands of queues with 0 or 1 items in them in rabbitmq like this:

$ sudo rabbitmqctl list_queues
Listing queues ...
c2e9b4beefc7468ea7c9005009a57e1d        1
1162a89dd72840b19fbe9151c63a4eaa        0
07638a97896744a190f8131c3ba063de        0
b34f8d6d7402408c92c77ff93cdd7cf8        1
f388839917ff4afa9338ef81c28aad75        0
8b898d0c7c7e4be4aa8007b38ccc00ea        1
3fb4be51aaaa4ac097af535301084b01        1

这似乎效率低下，但我进一步观察到，这些队列在处理完成后会持续很长时间.

This seems to be inefficient, but further I have observed that these queues persist long after processing is finished.

我发现似乎正在执行此操作的任务:

I have found the task that appears to be doing this:

@celery.task(ignore_result=True)
def write_pages(page_generator):  
    g = group(render_page.s(page) for page in page_generator)
    res = g.apply_async()

    for rendered_page in res:
        print rendered_page # TODO: print to file

似乎因为这些任务是在一个组中调用的，所以它们被扔到队列中但从未被释放.但是，我显然正在使用结果(因为当我遍历 res 时，我可以看到它们正在打印.所以，我不明白为什么这些任务会保留在队列中.

It seems that because these tasks are being called in a group, they are being thrown into the queue but never being released. However, I am clearly consuming the results (as I can view them being printed when I iterate through res. So, I do not understand why those tasks are persisting in the queue.

此外，我想知道正在创建的大量队列是否表明我做错了什么.

Additionally, I am wondering if the large number queues that are being created is some indication that I am doing something wrong.

感谢您对此的任何帮助！

Thanks for any help with this!

推荐答案

带有 AMQP 后端的 Celery 会将任务墓碑(结果)存储在以产生结果的任务 ID 命名的 AMQP 队列中.即使在结果耗尽后，这些队列仍将持续存在.

Celery with the AMQP backend will store task tombstones (results) in an AMQP queue named with the task ID that produced the result. These queues will persist even after the results are drained.

几个建议:

• 将 ignore_result=True 应用于您可以执行的每项任务.不要依赖其他任务的结果.
• 切换到不同的后端(也许 Redis - 无论如何它更有效):http://docs.celeryproject.org/en/latest/userguide/tasks.html

这篇关于为什么 celery 会向 rabbitmq 添加数千个队列，这些队列在任务完成后似乎仍然存在很长时间?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！