重试气流期间如何保持值 [英] How to persist a value during airflow retry
问题描述
每次dag触发任务时,我都想生成一个值并在重试期间保留该值(如果任务由于任何原因而无法运行)
Everytime a dag triggers a task I would like to generate a value and keep the value during retry (if the task fails to run for any reason)
我尝试在dag中声明一个变量(基本上将当前时间设置为该变量) 当任务失败并重试时,它会再次将变量设置为最新时间
I try to declare a variable inside the dag (basically set current time to the variable) When the task fails and it retries, it set the variable with latest time again
推荐答案
-
您可以使用BaseOperator的>
on_failure_callback
和on_retry_callback
参数将要保留的值作为xcom
You can make use of
on_failure_callback
andon_retry_callback
params ofBaseOperator
to push the value that you want to persist asxcom
此后,您可以在您的
Airflow
部署的任何DAG
中的任何task
中,从Xcom
检索数据,包括相同或不同的task
Thereafter, you can retrieve that data from
Xcom
in anytask
across anyDAG
of yourAirflow
deployment, including the retry attempt of same / differenttask
也看看
这篇关于重试气流期间如何保持值的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!