Django Celery:只执行一个长时间运行的进程的实例 [英] Django Celery: Execute only one instance of a long-running process

问题描述

我有一个长时间运行的进程，必须每五分钟运行一次，但不应同时运行多个进程实例.该过程通常不应超过五分钟，但我想确保第二个实例在运行结束时不会启动.

I have a long-running process that must run every five minutes, but more than one instance of the processes should never run at the same time. The process should not normally run past five min, but I want to be sure that a second instance does not start up if it runs over.

根据 上一页推荐，我正在使用 Django Celery 来安排这个长时间运行的任务.

Per a previous recommendation, I'm using Django Celery to schedule this long-running task.

我认为周期性任务不会起作用，因为如果我有 5 分钟的周期，如果另一个任务实例已经在运行，我不希望执行第二个任务.

I don't think a periodic task will work, because if I have a five minute period, I don't want a second task to execute if another instance of the task is running already.

我目前的实验如下:8:55，任务的一个实例开始运行.当任务完成时，它将触发另一个自身实例在接下来的五分钟标记处运行.因此，如果第一个任务在 8:57 完成，第二个任务将在 9:00 运行.如果第一个任务运行时间很长并在 9:01 完成，它将安排下一个实例在 9:05 运行.

My current experiment is as follows: at 8:55, an instance of the task starts to run. When the task is finishing up, it will trigger another instance of itself to run at the next five min mark. So if the first task finished at 8:57, the second task would run at 9:00. If the first task happens to run long and finish at 9:01, it would schedule the next instance to run at 9:05.

除了下面的简单示例之外，我一直在努力解决各种神秘的错误，而且我还没有找到任何其他关于人们从之前的实例调度任务的示例.我想知道是否有更好的方法来做我想做的事情.我知道有一种方法可以命名一个人的任务；也许有一种方法可以搜索具有相同名称的正在运行或计划的实例?关于每五分钟运行一次任务，但要确保一次只运行一个任务，是否有人有任何建议?

I've been struggling with a variety of cryptic errors when doing anything more than the simple example below and I haven't found any other examples of people scheduling tasks from a previous instance of itself. I'm wondering if there is maybe a better approach to doing what I am trying to do. I know there's a way to name one's tasks; perhaps there's a way to search for running or scheduled instances with the same name? Does anyone have any advice to offer regarding running a task every five min, but ensuring that only one task runs at a time?

谢谢你，乔

在 mymodule/tasks.py 中:

import datetime
from celery.decorators import task 

@task 
def test(run_periodically, frequency):

    run_long_process()
    now = datetime.datetime.now()
    # Run this task every x minutes, where x is an integer specified by frequency
    eta = (
      now - datetime.timedelta(
         minutes =  now.minute % frequency , seconds = now.second, 
         microseconds = now.microsecond ) ) + datetime.timedelta(minutes=frequency) 
    task = test.apply_async(args=[run_periodically, frequency,], eta=eta)  

来自 ./manage.py shell:

from mymodule import tasks
result = tasks.test.apply_async(args=[True, 5])

推荐答案

您可以使用与特殊锁配对的周期性任务，以确保任务一次执行一个.以下是 Celery 文档中的示例实现:

You can use periodic tasks paired with a special lock which ensures the tasks are executed one at a time. Here is a sample implementation from Celery documentation:

http:///ask.github.com/celery/cookbook/tasks.html#ensuring-a-task-is-only-executed-one-at-a-time

如果其中一个任务发生故障，您所描述的从上一次执行中调度任务的方法可以停止任务的执行.

Your described method with scheduling task from the previous execution can stop the execution of tasks if there will be failure in one of them.

这篇关于Django Celery:只执行一个长时间运行的进程的实例的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！