使用 Celery 动态创建队列 [英] Creating queues dynamically with Celery

问题描述

我正在使用 Django、Celery 和 RabbitMQ 编写一个邮件列表管理器.当有消息进来时，会为每个接收者执行一个任务.所有任务都进入一个队列，一个或多个工作人员使用队列中的任务，构建电子邮件消息并发送它们.

I am writing a mailing-list manager using Django, Celery, and RabbitMQ. When a message comes in, a task is executed for each recipient. All tasks go to a single queue, and one or more workers consume tasks from the queue, constructing the email messages and sending them.

单个队列会导致公平性问题:如果一条消息进入一个大型邮件列表，则会将大量任务添加到队列中，并且其他消息无法通过其他较小的邮件列表，直到所有消息都到达大名单已发送.我怎样才能找到解决这个问题的方法?

The single queue causes a fairness problem: if a message comes in to a large mailing list, a large number of tasks are added to the queue, and other messages cannot get through to other, smaller mailing lists until all the messages to the large list have been sent. How can I find a way around this?

从概念上讲，一种解决方案是为每个邮件列表创建一个队列，并让工作人员从各个队列循环中消费任务.考虑到我需要能够动态创建新的邮件列表，这在 Celery 中是否可行?我还没有看到动态创建队列或使工作人员从新队列中消费的功能.

Conceptually, a solution would be to create a queue for each mailing list and have the worker(s) consume tasks from the various queue round robin. Is this possible in Celery, given that I need to be able to create new mailing lists dynamically? I have not seen functionality for creating queues dynamically or for making the worker(s) consume from new queues.

推荐答案

如下图所示，考虑一个使用主题交换而不是直接交换的系统.

As shown in the image below consider a system which makes use of a topic exchange instead of a direct exchange.

与直接交换不同，主题交换允许我们将不同的消息路由到不同的队列.这是通过为每条消息设置 routing_key 并绑定某些队列以仅接受具有特定路由键的消息来完成的.

Unlike a direct exchange, Topic exchange allows us to route different messages to different queues. this is done by setting routing_key for each message and binding certain queues to accept only messages with a particular routing keys.

可以设置一个系统，将高优先级发送到专用队列和消费者，类似地，正常或低优先级消息由一个或多个队列处理.

A system can be set up that high priority go to a dedicated queue and consumer, similarly normal or low priority messages are handled by one or more queues.

Celery 支持跨多个工作人员的负载平衡(工作人员的数量取决于硬件).像 BROKER_POOL_LIMIT、PREFETCH_LIMIT 等 Celery 配置设置有助于更好地平衡负载并减少拥塞.

Celery supports load balancing across multiple workers(number of workers are hardware dependent). Celery config setting like BROKER_POOL_LIMIT, PREFETCH_LIMIT etc help better load balance and reduce congestion.

这篇关于使用 Celery 动态创建队列的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！