Redis芹菜的任务优先级 [英] Task priority in celery with redis

问题描述

我想用芹菜实现一个分布式的工作执行系统。鉴于RabbitMQ不支持优先级，而我非常需要此功能，因此我转向celery + redis。

I would like to implement a distributed job execution system with celery. Given that rabbitMQ doesn't support priorities and I'm painfully needing this feature, I turned to celery+redis.

在我的情况下，任务与硬件密切相关例如，任务A只能在Worker 1上运行，因为只有Worker 1的PC拥有必需的硬件。我将每个工作人员的CONCURRENCY设置为1，以便每个工作人员每次只能运行一个任务。每个任务大约需要2分钟。

In my situation, the tasks are closely related to hardware, for example, task A could only run on Worker 1 since only the PC of Worker 1 has got the necessary hardware. I set the CONCURRENCY of each worker to 1 so that a worker will only run one task each time. Each task takes about 2 minites.

要实现优先级功能，我首先尝试在添加 priority 参数时调用 apply_async（），例如 apply_async（priority = 0）和 apply_async（priority = 9）。在此测试中，我仅启动了一个COCURRENCY = 1的Worker，并以不同的优先级逐项启动了10项任务。我希望看到 apply_async（priority = 0）启动的任务会优先运行，但不幸的是，它们只是作为启动顺序启动的。

To implement the priority feature, first of all I tried adding priority argument when calling apply_async(), for example apply_async(priority=0) and apply_async(priority=9). In this test I launched only one Worker with COCURRENCY=1, and kicked off 10 tasks one by one with different priorities. I expected to see the tasks kicked off by apply_async(priority=0) will run in priority, but unfortunately they're just started as the kicking-off order.

然后我尝试进行一些处理。我克隆了每个任务，因此每个任务都有task_high和task_low，分别由 @ celery.task（priority = 0）和 @ celery.task装饰（优先级= 1）。然后我进行了与上述相同的测试，这次比较好，当启动顺序为 HH-LLLL-HHHH时，实际顺序为 HH-L-H-H-L-H-L-L-H。我想redis在这里做了一些计划和平衡工作。

Then I try to do some work around. I cloned each task, so for each one I have task_high and task_low, decorated by @celery.task(priority=0) and @celery.task(priority=1). Then I did the same test as above, this time it was better, when the kicking-off order is "HH-LLLL-HHHH", the real order comes out to be "HH-L-H-H-L-H-L-L-H". I suppose redis did some scheduling and balancing work here.

但这仍然不能满足我的期望。我希望得到一个像 HHHHHH-LLLL这样的命令，因为对于某些任务，我只有一台具有必要硬件的适当机器，并且希望高优先级的任务能够尽快运行。

But this still can't meet my expectation. I hope to get an order like "HHHHHH-LLLL", because for some tasks I have only one proper machine with the necessary hardware and hope the high-priority task to run as soon as possible.

我在Internet上搜索了其他工作，例如使用两个队列，一个队列用于高优先级任务，另一个队列用于低优先级，对于前一个使用2台机器，对于前一个使用1台机器。后者。但是，由于我的硬件非常有限，因此这对我不起作用。

I've searched for other work around on the Internet, for example using two queues, one for high-priority tasks and the other for low-priority, and using 2 machines for the former and 1 machine for the latter. But since my hardware is quite limited, this doesn't work for me.

请问您有什么建议吗？

推荐答案

Celery Redis传输确实支持优先级字段
，但是Redis本身没有优先级的概念。

The Celery Redis transport does honor the priority field, but Redis itself has no notion of priorities.

通过为每个队列
创建n个列表并在BRPOP命令中使用该顺序来实现优先级支持。
我在这里说 n 是因为即使有10（0-9）个优先级，默认情况下也将这些
合并为4个级别以节省资源。
这意味着名为 celery 的队列实际上将分为4个队列：

The priority support is implemented by creating n lists for each queue and using that order in the BRPOP command. I say n here because even though there are 10 (0-9) priority levels, these are consolidated into 4 levels by default to save resources. This means that a queue named celery will really be split into 4 queues:

['celery0', 'celery3`, `celery6`, `celery9`]

如果您想获得更高的优先级，可以设置 priority_steps 传输选项：

If you want more priority levels you can set the priority_steps transport option:

BROKER_TRANSPORT_OPTIONS = {
    'priority_steps': list(range(10)),
}

也就是说，请注意，这永远不会像在服务器级别实现的优先级
那样好，并且充其量可能是最好的。但是对于您的应用程序来说，
可能仍然足够。

That said, note that this will never be as good as priorities implemented at the server level, and may be approximate at best. But it may still be good enough for your application.

这篇关于Redis芹菜的任务优先级的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！