芹菜-在其他任务结束时安排定期任务 [英] Celery - Schedule periodic task at the end of another task

问题描述

我想在另一组任务结束时动态地与Celery安排一个定期任务.

I want to schedule a periodic task with Celery dynamically at the end of another group of task.

我知道如何使用Celery创建(静态)定期任务:

I know how to create (static) periodic tasks with Celery:

CELERYBEAT_SCHEDULE = {
      'poll_actions': {
          'task': 'tasks.poll_actions',
          'schedule': timedelta(seconds=5)
      }
}

但是我想从任务中动态创建定期作业(也许有一种方法可以在某些条件实现时(所有任务都已完成)停止这些定期作业.

But I want to create periodic jobs dynamically from my tasks (and maybe have a way to stop those periodic jobs when some condition is achieved (all tasks done).

类似的东西:

@celery.task
def run(ids):
    group(prepare.s(id) for id in ids) | execute.s(ids) | poll.s(ids, schedule=timedelta(seconds=5))

@celery.task
def prepare(id):
    ...

@celery.task
def execute(id):
    ...

@celery.task
def poll(ids):
    # This task has to be schedulable on demand
    ...

推荐答案

对此的直接解决方案要求您能够即时添加/删除Beat Scheduler条目.截至该问题的答案...

The straightforward solution to this requires that you be able to add/remove beat scheduler entries on the fly. As of the answering of this question...

如何动态添加/删除定期任务到芹菜(celerybeat)

这是不可能的.我怀疑它是否可以在此过渡中使用，因为...

This was not possible. I doubt it has become available in the interim because ...

您在这里混淆了两个概念.事件驱动工作"的概念和批处理时间表驱动的工作"(实际上只是事件按时间表发生的第一种情况).如果您真的考虑在这里做什么，就会发现存在相当复杂的边缘情况.从本质上来说，消息是分布式的，当从不同消息派生的组开始创建冲突条目时会发生什么?当您发现自己在预先安排的卡夫特山下时会怎么做?

You are conflating two concepts here. The notion of "Event Driven Work" and the idea of "Batch Schedule Driven Work"(which is really just the first case where the event happens on a schedule). If you really consider what you are doing here you'll find that there is a rather complex set of edge cases. Messages are distributed in nature what happens when groups spawned from different messages start creating conflicting entries? What do you do when you find yourself under a mountain of previously scheduled kruft?

使用消息传递系统时，您实际上是在构建递归树.做某事并产生更多消息以做更多事情的工作主轴.除了这些周期外，这些周期最终会达到其基本情况并终止.

When working with messaging systems you are really looking to build recursive trees. Spindles of work that do something and spawn more messages to do more things. Cycles(intended or otherwise) aside these ultimately achieve their base cases and terminate.

要真正实现的目标，答案就在于在消息传递系统和异步工作框架的限制范围内重新编码问题.

The answer to whatever you are actually trying to achieve lies with re-encoding your problem within the limitations of your messaging system and asynchronous work framework.

这篇关于芹菜-在其他任务结束时安排定期任务的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！