Celery-在一台服务器上运行不同的工作程序 [英] Celery - run different workers on one server

问题描述

我有2种任务：
Type1-一些高优先级的小任务。
Type2-很多优先级较低的繁重任务。

I have 2 kind of tasks : Type1 - A few of high priority small tasks. Type2 - Lot of heavy tasks with lower priority.

最初，我使用默认路由进行了简单的配置，没有使用路由键。这还不够-有时所有工作人员都在忙于Type2任务，因此Task1被延迟了。
我添加了路由键：

Initially i had simple configuration with default routing, no routing keys were used. It was not sufficient - sometimes all workers were busy with Type2 Tasks, so Task1 were delayed. I've added routing keys:

CELERY_DEFAULT_QUEUE = "default"
CELERY_QUEUES = {
    "default": {
        "binding_key": "task.#",
    },
    "highs": {
        "binding_key": "starter.#",
    },
}
CELERY_DEFAULT_EXCHANGE = "tasks"
CELERY_DEFAULT_EXCHANGE_TYPE = "topic"
CELERY_DEFAULT_ROUTING_KEY = "task.default"

CELERY_ROUTES = {
        "search.starter.start": {
            "queue": "highs",
            "routing_key": "starter.starter",
        },
}

所以现在我有2个队列-具有高优先级和低优先级的任务。

So now i have 2 queues - with high and low priority tasks.

问题是-如何使用不同的并发设置启动两个celeryd？

Problem is - how to start 2 celeryd's with different concurrency settings?

以前，celery在守护程序模式下使用（根据对此），因此只能以 / etc /开始init.d / celeryd start 是必需的，但是现在我必须运行2个具有不同队列和并发性的celeryds。我该怎么办？

Previously celery was used in daemon mode(according to this), so only start of /etc/init.d/celeryd start was required, but now i have to run 2 different celeryds with different queues and concurrency. How can i do it?

推荐答案

似乎答案-celery-multi-目前没有很好的文献记录。

It seems answer - celery-multi - is currently not documented well.

我需要的可以通过以下命令完成：

What I needed can be done by the following command:

celeryd-multi start 2 -Q:1 default -Q:2 starters -c:1 5 -c:2 3 --loglevel=INFO --pidfile=/var/run/celery/${USER}%n.pid --logfile=/var/log/celeryd.${USER}%n.log

我们要做的是启动2名工人，监听不同并发性的不同队列（默认为-Q：1，Q：2为启动器）-c：1 5 -c：2 3

What we do is starting 2 workers, which are listening to different queues (-Q:1 is default, Q:2 is starters ) with different concurrencies -c:1 5 -c:2 3

这篇关于Celery-在一台服务器上运行不同的工作程序的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！