Celery：任务层次结构后的回调 [英] Celery: Callback after task hierarchy

问题描述

我使用来自webapp的Celery启动任务层次结构。

任务

m使用以下任务：

• task_a


• task_b


• task_c


• notify_user

Django视图开始多个 task_a 实例。它们每个都进行一些处理，然后启动几个 task_b 实例。其中每个都进行一些处理，然后启动几个 task_c 实例。

要显示：

< h2>目标

我的目标是执行所有任务，并在整个层次结构完成后运行一个回调函数。

1. 视图应该只是启动任务，然后返回。


2. 每个子任务取决于父任务。父任务不直接依赖于子任务。


3. 只要父任务在子任务开始之前运行，所有操作都可以并行化。


4. 所有任务完成后，应调用 notify_user 回调函数。


5. notify_user 回调函数需要访问 task_c 的数据。

所有任务都应该是非阻塞的，因此 task_b 不应该等待所有

div解决方案原来是这个pull请求中提供的动态任务功能： https://github.com/celery/celery/pull/817 。这样，每个任务可以返回一组子任务，然后将替换队列中的原始taks。

I'm using Celery from a webapp to start a task hierarchy.

Tasks

I'm using the following tasks:

• task_a
• task_b
• task_c
• notify_user

A Django view starts several task_a instances. Each of them does some processing and then starts several task_b instances. And each of those does some processing and then starts several task_c instances.

To visualize:

Goals

My goal is to execute all the tasks, and to run a callback function as soon as the entire hierarchy has finished. Additionally, I want to be able to pass data from the lowest tasks to the top level.

1. The view should just "kick off" the tasks and then return.
2. Each subtask depends on the parent task. The parent task does not depend directly on the child task. After a parent task has started all the child tasks, it can be stopped.
3. Everything can be parallelized, as long as the parent task runs before the child task is started.
4. After all the tasks have finished, the notify_user callback function should be called.
5. The notify_user callback function needs access to data from the task_cs.

All the tasks should be non-blocking, so task_b should not wait for all the task_c subtasks to finish.

What would be the right way to achieve the goal stated above?

解决方案

The solution turned out to be the dynamic task feature provided in this pull request: https://github.com/celery/celery/pull/817. With this, each task can return a group of subtasks, which will then replace the original taks in the queue.

这篇关于Celery：任务层次结构后的回调的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！