芹菜不使用所有并发插槽 [英] Celery not using all concurrent slots

问题描述

我有一个Celery群集，该群集由配备8核处理器的计算机组成。每台计算机上都有一个工人，并发因子设置为8（-c8）。

我经常看到节点上有很多保留任务，但是只有一个或两个同时运行。我的任务通常需要大量的计算和I / O才能长期运行。

关于为什么发生这种情况的任何想法，以及我可以做些什么来增加数量同时运行的任务？芹菜是否会根据系统负载限制活动任务的数量？我仔细阅读了文档，但内容简短。

解决方案

感谢香蕉，我认为我找到了答案。

I have a Celery cluster made up of machines with 8-core processors. Each machine has one worker that is set to a concurrency factor of 8 (-c8).

I often see nodes with a lot of reserved tasks, but only one or two are running simultaneously. My tasks are often long-running with a lot of compute and I/O.

Any ideas as to why this is happening, and what I can do to increase the number of tasks simultaneously running? Does celery throttle the number of active tasks based on system load? I looked through the documentation but came up short.

解决方案

Thanks to banana, I think I found the answer.

Some of my tasks were spawning subprocesses, which Celery counts in its concurrency.

这篇关于芹菜不使用所有并发插槽的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！