django + celery:禁用一名工作人员的预取功能，是否存在错误? [英] django + celery: disable prefetch for one worker, Is there a bug?
问题描述
我有一个芹菜的Django项目
I have a Django project with celery
由于RAM的限制，我只能运行两个工作进程.
Due to RAM limitations I can only run two worker processes.
我混合了慢"和快"任务.快速任务应尽快执行.在很短的时间内(0.1s-3s)可以有许多快速任务，因此理想情况下，两个CPU都应处理它们.
I have a mix of 'slow' and 'fast' tasks. Fast tasks shall be executed ASAP. There can be many fast tasks in a short time frame (0.1s - 3s), so ideally both CPUs should handle them.
缓慢的任务可能会运行几分钟，但结果可能会延迟.
Slow tasks might run for a few minutes but the result can be delayed.
慢任务的发生频率较低，但是可能同时出现2或3个队列.
Slow tasks occur less often, but it can happen that 2 or 3 are queued up at the same time.
我的想法是拥有一个:
- 1个并发性为1的芹菜工人W1，仅处理快速任务
- 1个并发性为1的芹菜工人W2，可以处理快速任务和慢速任务.
celery默认情况下具有任务预取乘数(https://docs.celeryproject.org/zh-CN/latest/userguide/configuration.html#worker-prefetch-multiplier )，即4，这意味着4个快速任务可能会排在慢速任务之后，并且可能会延迟几分钟.因此，我想为工作者W2禁用预取.该文档指出:
celery has by default a task prefetch multiplier ( https://docs.celeryproject.org/en/latest/userguide/configuration.html#worker-prefetch-multiplier ) of 4, which means that 4 fast tasks could be queued behind a slow task and could be delayed by several minutes. Thus I'd like to disable prefetch for worker W2. The doc states:
要禁用预取，请将worker_prefetch_multiplier设置为1.设置为0将允许工人继续消耗尽可能多的食物消息.
To disable prefetching, set worker_prefetch_multiplier to 1. Changing that setting to 0 will allow the worker to keep consuming as many messages as it wants.
但是我观察到的是，prefetch_multiplier为1时，一个任务被预取，并且仍然会因执行缓慢的任务而延迟.
However what I observe is, that with a prefetch_multiplier of 1 one task is prefetched and would still be delayed by a slow task.
这是文档错误吗?这是实现错误吗?还是我误解了文档?有什么方法可以实现我想要的吗?
Is this a documentation bug? Is this an implementation bug? Or do I misunderstand the documentation? Is there any way to implement what I want?
我执行以启动工作程序的命令是:
The commands, that I execute to start the workers are:
celery -A miniclry worker --concurrency=1 -n w2 -Q=fast,slow --prefetch-multiplier 0
celery -A miniclry worker --concurrency=1 -n w1 -Q=fast
我的芹菜设置为默认设置，除了:
my celery settings are default except:
CELERY_BROKER_URL = "pyamqp://*****@localhost:5672/mini"
CELERY_TASK_ROUTES = {
'app1.tasks.task_fast': {"queue": "fast"},
'app1.tasks.task_slow': {"queue": "slow"},
}
我的django项目的celery.py文件是:
my django project's celery.py file is:
from __future__ import absolute_import
import os
from celery import Celery
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'miniclry.settings')
app = Celery("miniclry", backend="rpc", broker="pyamqp://")
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()
我的django项目的 __ init __.py 是
The __init__.py of my django project is
from .celery import app as celery_app
__all__ = ('celery_app',)
我的工人的密码
import time, logging
from celery import shared_task
from miniclry.celery import app as celery_app
logger = logging.getLogger(__name__)
@shared_task
def task_fast(delay=0.1):
logger.warning("fast in")
time.sleep(delay)
logger.warning("fast out")
@shared_task
def task_slow(delay=30):
logger.warning("slow in")
time.sleep(delay)
logger.warning("slow out")
如果我从管理外壳执行以下操作，则仅在慢速任务完成后才执行一个快速任务.
If I execute following from a management shell I see, that one fast task is only executed after the slow task finished.
from app1.tasks import task_fast, task_slow
task_slow.delay()
for i in range(30):
task_fast.delay()
有人可以帮忙吗?
如果认为有帮助，我可以发布整个测试项目.只是建议有关交换此类项目的建议SO方法
I could post the entire test project if this is considered helpful. Just advise about the recommended SO way of exchanging such kind of projects
版本信息:
- celery == 4.3.0
- Django == 1.11.25
- Python 2.7.12
推荐答案
I confirm the issue, there is a bug in this section of the documentation. worker_prefetch_multiplier = 1 will just as it says, set the worker's prefetch to 1, means worker will hold one more task in addition to one that is executing at the moment.
要实际禁用预取，还需要使用 task_acks_late = True 以及预取设置，请参见
To actually disable the prefetch you also need to use task_acks_late = True along with the prefetch setting, see this docs section
这篇关于django + celery:禁用一名工作人员的预取功能，是否存在错误?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
