重用气流中的任务 [英] Reuse tasks in airflow

问题描述

我正在尝试通过气流来编排一些数据管道.每个摄取管道都有多个任务.这些任务在多个摄取管道中不断重复.如何在DAGS中重用气流中的任务?

I'm trying out airflow for orchestrating some of my data pipelines. I'm having multiple tasks for each ingestion pipeline. The tasks are getting repeated across multiple ingestion pipelines. How can I reuse a task across DAGS in airflow?

推荐答案

• 就像 object是class 的实例，Airflow 任务是 BaseOperator )
• 因此，编写一个可重用"(也称为通用)运算符，并通过传递不同的参数(尤其是task_id)，在整个管道中使用它100次.
• Just like object is an instance of a class, an Airflow task is an instance of an Operator (strictly speaking, BaseOperator)
• So write a "re-usable" (aka generic) operator and use it 100s of times across your pipeline(s) simply by passing different params (particularly task_id)

这篇关于重用气流中的任务的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！