通过命令行将GCP凭据添加到气流中 [英] Add GCP credentials to airflow via command line

问题描述

气流允许我们通过命令行气流连接添加连接信息。这可以帮助通过ansible或其他开发者工具自动部署气流装置。

Airflow allows us to add connection information via command-line airflow connections. This can help with automated deployment of airflow installations via ansible or other dev-ops tools.

目前尚不清楚如何向Google Cloud Platform（服务帐户）添加连接通过命令行ariflow。

It is unclear how connections to google cloud platform (service accounts) can be added to ariflow via command line.

推荐答案

在airflow 1.9之前，以下示例概述了如何使用DAG添加连接信息： https://gist.github.com/yu-iskw/42f9f0aa6f2ff0a2a375d43881e13b49

Pre airflow 1.9 the following example outlines how to use a DAG to add connection information: https://gist.github.com/yu-iskw/42f9f0aa6f2ff0a2a375d43881e13b49

def add_gcp_connection(ds, **kwargs):
    """"Add a airflow connection for GCP"""
    new_conn = Connection(
        conn_id=<CONNECTION_ID>,
        conn_type='google_cloud_platform',
    )
    scopes = ['https://www.googleapis.com/auth/cloud-platform']
    conn_extra = {
        "extra__google_cloud_platform__scope": ",".join(scopes),
        "extra__google_cloud_platform__project":
            "<GCP_PROJECT_NAME>",
        "extra__google_cloud_platform__key_path":
            "<GCP_CREDENTIALS_ABSOLUTE_PATH.json>"
    }
    conn_extra_json = json.dumps(conn_extra)
    new_conn.set_extra(conn_extra_json)

    session = settings.Session()
    session.add(new_conn)
    session.commit()

从气流1.9向前可以：

From airflow 1.9 forward one can:

airflow connections -a \
  --conn_id=<CONNECTION_ID> \
  --conn_type=google_cloud_platform \
  --conn_extra='{ "extra__google_cloud_platform__key_path":" '`
        `'<GCP_CREDENTIALS_ABSOLUTE_PATH.json>", '`
    `'"extra__google_cloud_platform__project": '`
        `'"<GCP_PROJECT_NAME>", '`
    `'"extra__google_cloud_platform__scope":  '`
        `'"https://www.googleapis.com/auth/cloud-platform"}'

这篇关于通过命令行将GCP凭据添加到气流中的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！