Celery一次又一次地运行长时间运行的已完成任务 [英] Celery is rerunning long running completed tasks over and over

问题描述

我有一个python celery-redis队列，一次处理上传和下载有价值的数据。

I've a python celery-redis queue processing uploads and downloads worth gigs and gigs of data at a time.

很少有上传需要花费几个小时的时间。但是，一旦完成了这样的任务，我就目睹了这种奇怪的芹菜行为，即芹菜调度程序通过将其再次发送给工作人员（我正在运行一个工作人员）来重新运行刚刚完成的任务，并且它在同一时间发生了2次任务！

Few of the uploads takes upto few hours. However once such a task finishes, I'm witnessing this bizarre celery behaviour that the celery scheduler is rerunning the just concluded task again by sending it again to the worker (I'm running a single worker) And it just happened 2times on the same task!

有人可以帮助我知道为什么会发生这种情况以及如何预防吗？

Can someone help me know why is this happening and how can I prevent it?

任务肯定是干净完成的，没有错误报告，因为这些都是运行时间非常长的任务。

The tasks are definitely finishing cleanly with no errors reported just that these are extremely long running tasks.

推荐答案

我最近遇到了这个问题，最终发现由于组合，任务多次运行

的任务预取，并且任务超出了
可见性超时。任务会在执行之前得到确认（除非您设置ACKS_LATE = True），
和默认情况下每个进程会预取4个任务。第一个任务将在执行前确认
，但是如果执行需要一个多小时，则
个其他预取的任务将被交付给另一个工作人员，在该处另外
个执行时间（或者在您的情况下，
由同一工作人员执行了额外的时间）。

I recently ran into this issue, and eventually figured out that tasks were running multiple times because of a combination of task prefetching and tasks exceeded the visibility timeout. Tasks are acknowledged right before they're executed (unless you set ACKS_LATE=True), and by default 4 tasks are prefetched per process. The first task will be acknowledged before execution, but if it takes over an hour to execute then the other prefetched tasks will be delivered to another worker where it will be executed an additional time (or in your case, executed an additional time by the same worker).

您可以通过将可见性超时时间增加到比任务的最长运行时间更长的时间来解决：

You can solve by increasing the visibility timeout to something longer than the longest possible runtime of your tasks:

BROKER_TRANSPORT_OPTIONS = {'visibility_timeout': 3600*10}  # 10 hours

您还可以设置 PREFETCH_MULTIPLIER = 1 禁用预取，以便长时间运行的任务不会阻止
其他任务被确认。

You could also set PREFETCH_MULTIPLIER=1 to disable prefetching so that long running tasks don't keep other tasks from being acknowledged.

这篇关于Celery一次又一次地运行长时间运行的已完成任务的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！