气流-无论上游成功/失败，都可以运行任务 [英] Airflow - run task regardless of upstream success/fail

问题描述

我有一个DAG，它可以并行地散布到多个独立单元。它在AWS中运行，因此我们有一些任务，这些任务可在DAG启动时将AutoScalingGroup扩展到最大数量的工作程序，在DAG完成时扩展到最小的工作程序数。简化的版本如下所示：

I have a DAG which fans out to multiple independent units in parallel. This runs in AWS, so we have tasks which scale our AutoScalingGroup up to the maximum number of workers when the DAG starts, and to the minimum when the DAG completes. The simplified version looks like this:

           | - - taskA - - |
           |               |
scaleOut - | - - taskB - - | - scaleIn
           |               |
           | - - taskC - - |

但是，并行集中的某些任务偶尔会失败，并且我无法获得scaleDown在任何AC任务失败时运行的任务。

However, some of the tasks in the parallel set fail occasionally, and I can't get the scaleDown task to run when any of the A-C tasks fail.

在所有其他任务都完成后（成功）在DAG末尾执行任务的最佳方法是什么？或失败）？ Depends_on_upstream设置听起来像我们需要的，但实际上并没有根据测试执行任何操作。

What's the best way to have a task execute at the end of the DAG, once all other tasks have completed (success or fail)? The depends_on_upstream setting sounded like what we needed, but didn't actually do anything based on testing.

推荐答案

所有运算符都有一个参数 trigger_rule 可以设置为'all_done'，无论前一个任务的失败或成功，都会触发该任务

All operators have an argument trigger_rule which can be set to 'all_done', which will trigger that task regardless of the failure or success of the previous task(s).

您可以将要运行的任务的触发规则设置为'all_done'而不是默认的'all_success'。

You could set the trigger rule for the task you want to run to 'all_done' instead of the default 'all_success'.

带有该参数的简单bash运算符任务看起来像：

A simple bash operator task with that argument would look like:

task = BashOperator(
    task_id="hello_world",
    bash_command="echo Hello World!",
    trigger_rule="all_done",
    dag=dag
    )

这篇关于气流-无论上游成功/失败，都可以运行任务的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！