Celery：具有并发和保留任务的工作者仅运行1个任务 [英] Celery: Worker with concurrency and reserved tasks only running 1 task

问题描述

代码中的某些任务执行时间越来越长。

Some of the tasks in my code were taking longer and longer to execute.

在检查时，我注意到尽管我将我的工作程序节点设置为并发6，并且存在执行工作的6个过程，但正在运行的任务下仅显示1个任务。这是一些视觉上的证明：

Upon inspection I noticed that although I have my worker node set to concurrency 6, and 6 processes exist to 'do work', but only 1 task is shown under 'running tasks'. Here is a little visual proof:

以下是工作程序选项：

这是该任务的选项卡仅具有1个正在运行的进程的工作进程：

And here is the task tab for that worker with only 1 running process:

我发现，如果重新启动celery，并发性将再次得到尊重，并且我将看到> 1个正在运行的任务，但是经过一定时间/任务后，恢复到这种行为。

I have found that if I restart celery, the concurrency is once again respected and i will see >1 running task, but after some amount of time/tasks it reverts back to this behavior..

有任何解决此间歇性问题的想法吗？

Any ideas for fixing this intermittent problem?

推荐答案

我不确定这是否是您的用例，但是当我混合使用长时任务和短时任务时，也会遇到类似的问题。基本上发生的事情是，某个进程可以在某个时刻启动一个运行时间很长的任务，同时预取一些其他任务，以防止其他进程消耗它们。因此，我禁用了预取功能，这仅在您执行许多简短任务时才有用。

I'm not sure if it's your use case, but I ran into similar problems when I had a mix of long and short tasks. Basically what happened is that at some point a process could start a very long running task, while prefetching a few other tasks, preventing them from being consumed by other processes. So I disabled the prefetching stuff, which is useful only if you're running a lot of short tasks.

要禁用预取功能，您需要使用Celery 3.1+和 Ofair设置，例如：

To disable the prefetch, you need Celery 3.1+ and the Ofair setting, for instance:

celery -A proj worker -l info -Ofair

这篇关于Celery：具有并发和保留任务的工作者仅运行1个任务的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！