如何在 Airflow 中使用 HashiCorp Vault? [英] How can one use HashiCorp Vault in Airflow?
问题描述
我开始使用 Apache Airflow,我想知道如何有效地使用存储在 Vault 中的机密和密码.不幸的是,除了 Airflow 中尚未实现的钩子之外,搜索不会返回有意义的答案 项目本身.
I am starting to use Apache Airflow and I am wondering how to effectively make it use secrets and passwords stored in Vault. Unfortunately, search does not return meaningful answers beyond a yet-to-be-implemented hook in Airflow project itself.
我总是可以使用 Python 的 hvac
模块从 PythonOperator
访问 Vault,但我想知道是否有任何更好的 方法或 良好实践(例如,我错过了一个 Airflow 插件).
I can always use Python's hvac
module to generically access Vault from PythonOperator
but I was wondering if there is any better way or a good practice (e.g. maybe an Airflow plugin I missed).
推荐答案
Airflow >=1.10.10 支持 Secrets Backends 并支持从 Hashicorp Vault 获取 Airflow 变量和连接.
Airflow >=1.10.10 supports Secrets Backends and supports getting Airflow Variables and Connections from Hashicorp Vault.
Airflow 文档中的更多详细信息:https://airflow.apache.org/docs/stable/howto/use-alternative-secrets-backend.html#hashicorp-vault-secrets-backend
More Details in Airflow Docs: https://airflow.apache.org/docs/stable/howto/use-alternative-secrets-backend.html#hashicorp-vault-secrets-backend
如果您想在本地进行测试,请查看 https://www.astronomer.io/guides/airflow-and-hashicorp-vault/
If you want to test it locally check the tutorial at https://www.astronomer.io/guides/airflow-and-hashicorp-vault/
在 airflow.cfg
中设置以下配置,根据您的环境进行更新:
Set the following config in airflow.cfg
, update based on your environment:
backend = airflow.contrib.secrets.hashicorp_vault.VaultBackend
backend_kwargs = {"connections_path": "connections", "variables_path": "variables", "mount_point": "airflow", "url": "http://127.0.0.1:8200"}
测试集成的 DAG 示例:
Example DAG to test the integration:
from airflow import DAG
from airflow.operators.python_operator import PythonOperator
from datetime import datetime
from airflow.hooks.base_hook import BaseHook
def get_secrets(**kwargs):
conn = BaseHook.get_connection(kwargs['my_conn_id'])
print(f"Password: {conn.password}, Login: {conn.login}, URI: {conn.get_uri()}, Host: {conn.host}")
with DAG('example_secrets_dags', start_date=datetime(2020, 1, 1), schedule_interval=None) as dag:
test_task = PythonOperator(
task_id='test-task',
python_callable=get_secrets,
op_kwargs={'my_conn_id': 'smtp_default'},
)
这篇关于如何在 Airflow 中使用 HashiCorp Vault?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!