气流：DAG标记为“成功”；如果一项任务失败，由于触发规则ALL_DONE [英] Airflow : DAG marked as "success" if one task fails, because of trigger rule ALL_DONE

问题描述

我有以下3个任务的DAG：

I have the following DAG with 3 tasks :

start --> special_task --> end

中间任务可以成功或失败，但结束 必须总是被执行（想象这是清理资源的任务）。为此，我使用了触发规则 ALL_DONE ：

The task in the middle can succeed or fail, but end must always be executed (imagine this is a task for cleanly closing resources). For that, I used the trigger rule ALL_DONE :

end.trigger_rule = trigger_rule.TriggerRule.ALL_DONE

如果在以下情况下正确执行 end 特殊任务失败。但是，由于 end 是最后一个任务并且成功完成，因此DAG始终标记为 SUCCESS 。

Using that, end is properly executed if special_task fails. However, since end is the last task and succeeds, the DAG is always marked as SUCCESS.

如何配置DAG，以便如果其中一项任务失败，则整个DAG都标记为 FAILED ？

How can I configure my DAG so that if one of the tasks failed, the whole DAG is marked as FAILED?

import datetime

from airflow import DAG
from airflow.operators.bash_operator import BashOperator
from airflow.utils import trigger_rule

dag = DAG(
    dag_id='my_dag',
    start_date=datetime.datetime.today(),
    schedule_interval=None
)

start = BashOperator(
    task_id='start',
    bash_command='echo start',
    dag=dag
)

special_task = BashOperator(
    task_id='special_task',
    bash_command='exit 1', # force failure
    dag=dag
)

end = BashOperator(
    task_id='end',
    bash_command='echo end',
    dag=dag
)
end.trigger_rule = trigger_rule.TriggerRule.ALL_DONE

start.set_downstream(special_task)
special_task.set_downstream(end)

这篇文章似乎相关，但是答案不符合我的需要，因为必须执行下游任务 end （因此，强制性 trigger_rule ）。

This post seems to be related, but the answer does not suit my needs, since the downstream task end must be executed (hence the mandatory trigger_rule).

推荐答案

为@JustinasMarozas 在评论，一种解决方案是创建一个虚拟任务，例如：

As @JustinasMarozas explained in a comment, a solution is to create a dummy task like :

dummy = DummyOperator(
    task_id='test',
    dag=dag
)

并将其下游绑定到 special_task ：

failing_task.set_downstream(dummy)

因此，DAG被标记为失败，并且 dummy 任务被标记为 upstream_failed 。

Thus, the DAG is marked as failed, and the dummy task is marked as upstream_failed.

希望在那里是一个开箱即用的解决方案，但请耐心等待，此解决方案可以完成工作。

Hope there is an out-of-the-box solution, but waiting for that, this solution does the job.

这篇关于气流：DAG标记为“成功”；如果一项任务失败，由于触发规则ALL_DONE的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！