如何在不使用 GUI 的情况下创建、更新和删除气流变量? [英] How to create, update and delete airflow variables without using the GUI?

问题描述

我一直在学习气流并为 ETL 管道编写 DAG.它涉及使用 AWS 环境(S3、Redshift).它处理在将数据存储在 redshift 之后将数据从一个存储桶复制到另一个存储桶.我将存储桶名称和前缀作为变量存储在气流中，您必须打开 GUI 并手动添加它们.

I have been learning airflow and writing DAGs for an ETL pipeline. It involves using the AWS environment (S3, Redshift). It deals with copying data from one bucket to another after storing it in redshift. I am storing bucket names and prefixes as Variables in airflow for which you have to open the GUI and add them manually.

在以下选项中，哪个是业内最安全和最广泛使用的做法

Which is the most safest and widely used practice in the industry out of the following options

• 我们可以使用 airflow.cfg 来存储我们的变量(bucket 名称)并在我们的 DAG 中访问它们吗?
• 使用自定义配置文件并使用 configparser
解析其内容
• 使用 GUI 添加变量

• Can we use airflow.cfg to store our variables (bucket names) and access them in our DAGs?
• Use a custom configuration file and parse its contents using configparser
• Use the GUI to add variables

推荐答案

总结一下:可以使用airflow cli从json文件中进行变量的导入操作.您可以使用以下命令 airflow variables -i[1] 并通过气流 CICD 管道构建它或手动运行它.那应该处理插入/更新情况.对于删除，您可以显式调用 airflow variables -x，我认为目前您不能在气流中进行批量删除.

To summary: you can use airflow cli to perform an import operation of variables from a json file. You could use the following command airflow variables -i[1] and build it via airflow CICD pipeline or manually run it. That should handle the insert/update case. For deletion, you can call airflow variables -x explicitly, I don't think currently you can do a batch delete in airflow now.

您可以让 JSON 文件看起来像以下格式，并带有键值:

You can have a JSON file looks like the following format with key value:

{
    "foo1": "bar1",
    "foo2": "bar2"
}

这里要注意一点:您可以将变量视为键值存储，因此请确保在导入时没有重复的键(否则您可能会以意外结果覆盖它)

One thing to note here: you can treat the variable as key-value storage, so make sure you don't have duplicated keys when you import (otherwise you might override it with unexpected result)

[1]气流.apache.org/cli.html#variables

[1] airflow.apache.org/cli.html#variables

这篇关于如何在不使用 GUI 的情况下创建、更新和删除气流变量?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！