芹菜预取乘数设置无效 [英] Celery prefetch multiplier setting no effect

问题描述

我有一系列运行多个工作程序的服务器.这些是长期运行的任务，需要10分钟到36个小时不等.因此，我希望尽可能避免预取，并让每个工作人员在完成后承担一项新的任务.

I have a series of servers running multiple workers. These are long-running tasks, requiring anywhere from 10 minutes to 36 hours. As such, I'd like to avoid prefetching if at all possible and have each worker pick up a singular new task after completing.

我正在使用 celeryd init.d服务，并且拥有

I'm using the celeryd init.d service, and have

CELERYD_ACKS_LATE = True
CELERYD_PREFETCH_MULTIPLIER = 1

在工作服务器的/etc/default/celeryd 中.

但是，如果我转到python shell控制台并执行

However, if I go to the python shell console and do

from work_project.celery import app
inspector = app.control.inspect()
inspector.stats()

我在字典输出中得到了这个

I get this in the dict output:

...
u'prefetch_count': 4,
...

我将RabbitMQ用作代理，将Redis用作后端.我怀疑这种预取会导致几个工作人员在完成初始任务后仍然闲置，因为其他工作人员在队列中有待处理的任务.例如，当前我有两台服务器，总共有九个工人在运行.在执行20个任务的批处理开始时，所有9个任务都同时运行.现在，大约在9000万后，只有六名工人处于活动状态.

I'm using RabbitMQ as the broker and Redis as the backend. I suspect this prefetching is having the effect of several workers remaining idle after completing their initial task because other workers have pending tasks in their queues. For example, currently I have two servers with nine workers total running. At the start of a 20 task batch, all nine were running concurrently. Now, approximately 90m later, only six workers are active.

推荐答案

我以这种方式处理相同的问题.

I deal with the same problem this way.

正在运行的工作者，其中: celery worker --prefetch-multiplier = 1 --concurrency = 1 -Ofair

Running worker with: celery worker --prefetch-multiplier=1 --concurrency=1 -Ofair

在Django设置下: CELERY_TASK_ACKS_LATE = True

Under Django settings: CELERY_TASK_ACKS_LATE = True

Django==3.0.2
celery==4.4.0

---

并发性= 1很重要，因为prefetch-multiplier会在并发性上相乘

---

It's important to have concurrency=1, because prefetch-multiplier multiplies on concurrency

-Ofair-可能是多余的，需要检查

-Ofair - might be redundant, need to check

对工人的迟到参数找不到我(或没有工作)

Acks late param on worker didn't find (or didn't work) for me

没有django的设置需要检查(对此太懒了).可能是 CELERYD_TASK_ACKS_LATE = True

Setting without django need to check (too lazy for it). Might be CELERYD_TASK_ACKS_LATE = True

这篇关于芹菜预取乘数设置无效的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！