如何设置一个变量仅在特定的dag_run期间使用 [英] How can one set a variable for use only during a certain dag_run

问题描述

如何设置用于特定dag_run的变量。我知道可以在xcom中设置值，但并不是我使用的所有运算符都具有xcom支持。我也不想将值存储到Variables数据存储中，以防在当前运行时开始另一次dag运行而需要存储其他值。

How do I set a variable for use during a particular dag_run. I'm aware of setting values in xcom, but not all the operators that I use has xcom support. I also would not like to store the value into the Variables datastore, in case another dag run begins while the current one is running, that need to store different values.

推荐答案

问题尚不清楚，但是根据我的推断，我将尽力消除您的疑虑

The question is not clear, but from whatever I can infer, I'll try to clear your doubts

并非我使用的所有运营商都提供xcom支持

not all the operators that I use has xcom support

显然您已经将 xcom 与其他构造误认为是因为 xcom 功能是 TaskInstance 的一部分以及功能< a href = https://github.com/apache/airflow/blob/v1-10-stable/airflow/models.py#L3159 rel = nofollow noreferrer> xcom_push（） 和 xcom_pull（） 是在 BaseOperator 本身（是所有 Airflow 操作符 s）

Apparently you've mistaken xcom with some other construct because xcom feature is part of TaskInstance and the functions xcom_push() and xcom_pull() are defined in BaseOperator itself (which is the parent of all Airflow operators)

我也不希望将值存储到Variables数据存储区中，如果在当前运行时另一次dag运行开始时
表示
需要存储其他值。

I also would not like to store the value into the Variables datastore, in case another dag run begins while the current one is running, that need to store different values.

很容易（而且很容易）将分离出的变量 - DAG 基础（请参阅点（6））；但是可以，对于单个 DAG 的不同 DagRun ，这种隔离将是一个挑战。我可以认为 xcom 是最简单的解决方法。请查看此，以了解 Xcom s的用法。 。

It is straightforward (and no-brainer) to separate-out Variables on per-DAG basis (see point (6)); but yes for different DagRuns of a single DAG, this kind of isolation would be a challenge. I can think of xcom to be the easiest workaround for this. Have a look at this for some insights on usage of Xcoms.

此外，如果要操作变量 s （或任何其他 Airflow 模型）在运行时 （尽管我建议您避免尤其是对于 Variable s）， Airflow 也提供了完全自由地利用底层的 SQLAlchemy ORM 框架。您可以从此中获得启发。

Additionally, if you want to manipulate Variables (or any other Airflow model) at runtime (though I would recommend you to avoid it particularly for Variables), Airflow also gives completely liberty to exploit the underlying SQLAlchemy ORM framework for that. You can take inspiration from this.

这篇关于如何设置一个变量仅在特定的dag_run期间使用的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！