Cloud Composer + Airflow:设置DAW以在HTTP上触发(或者我应该使用Cloud Functions吗?) [英] Cloud Composer + Airflow: Setting up DAWs to trigger on HTTP (or should I use Cloud Functions?)

问题描述

最终，我想做的是有一个Python脚本，该脚本可以在创建HTTP请求时动态地运行.就像这样:App 1运行并发出一个Webhook，Python脚本立即捕获该Webhook并执行其所有操作.

Ultimately, what I want to do is have a Python script that runs whenever a HTTP request is created, dynamically. It'd be like: App 1 runs and sends out a webhook, Python script catches the webhook immediately and does whatever it does.

我看到您可以在GCP中使用Composer和Airflow来做到这一点.

I saw that you could do this in GCP with Composer and Airflow.

但是在这些工具之后，我遇到了几个问题 https://cloud.google.com/composer/docs/how-to/using/triggering-with-gcf :

But I'm having several issues following these instrutions https://cloud.google.com/composer/docs/how-to/using/triggering-with-gcf:

1. 在Cloud Shell中运行此操作以授予Blob签名权限:

1. Running this in Cloud Shell to grant blob signing permissions:

gcloud iam服务帐户add-iam-policy-binding
[e-mail]
--member = serviceAccount:[e-mail]
--role = roles/iam.serviceAccountTokenCreator

gcloud iam service-accounts add-iam-policy-binding
[e-mail]
--member=serviceAccount:[e-mail]
--role=roles/iam.serviceAccountTokenCreator

当我输入项目ID时，我会得到一个找不到针对您的[e-mail]的盖亚ID"

When I put in my project ID, I get a "Gaia id not found for [e-mail]"

1. 当我运行airflow_uri = environment_data ['config'] ['airflowUri']位时，我在'config'上遇到关键错误.

---

是否有更好的方法来做我想做的事情(即动态运行Python脚本)?

---

Is there a better way to do what I'm trying to do (i.e. run Python scripts dynamically)?

推荐答案

获得原因:未找到电子邮件的盖亚ID< project-id> @ appspot.gserviceaccount.com 错误在您的项目中启用所有必需的API.请按照以下步骤操作:

The reason for getting: Gaia id not found for email <project-id>@appspot.gserviceaccount.com error is not enabling all needed APIs in your project. Please follow the steps:

1. 创建或选择您要使用的Google Cloud Platform项目.
2. 启用Cloud Composer，Google Cloud Functions和Cloud Identity以及Google Identity and Access Management(IAM)API.您可以在 Menu ->中找到它.产品-> Marketplace ，然后输入相应API的名称.
3. 授予对Cloud Functions Service帐户的Blob签名权限.为了使GCF向保护气流Web服务器的代理Cloud IAP进行身份验证，您需要向Appspot服务帐户GCF授予服务帐户令牌创建者角色.为此，请在您的Cloud Shell中运行以下命令，将项目名称替换为< your-project-id> :

1. Create or select Google Cloud Platform Project you wish to work with.
2. Enable the Cloud Composer, Google Cloud Functions and Cloud Identity and Google Identity and Access Management (IAM) APIs. You can find it in Menu -> Products -> Marketplace and typing the name of corresponding API.
3. Grant blob signing permissions to the Cloud Functions Service Account. In order for GCF to authenticate to Cloud IAP, the proxy that protects the Airflow webserver, you need to grant the Appspot Service Account GCF the Service Account Token Creator role. Do so by running the following command in your Cloud Shell, substituting the name of your project for <your-project-id>:

gcloud iam service-accounts add-iam-policy-binding \
<your-project-id>@appspot.gserviceaccount.com \
--member=serviceAccount:<your-project-id>@appspot.gserviceaccount.com \
--role=roles/iam.serviceAccountTokenCreator

我测试了该方案，首先没有启用API，并且检索到与您相同的错误.启用API后，错误消失并且IAM策略已正确更新.

I tested the scenario, firstly without enabling APIs and I've retrieved the same error as you. After enabling the APIs, error disappear and IAM policy has been updated correctly.

已经充分描述了代码实验室教程，其中显示了使用Google Cloud Functions触发DAG的工作流程.

There is already well described Codelabs tutorial, which shows the workflow of triggering the DAG with Google Cloud Functions.

这篇关于Cloud Composer + Airflow:设置DAW以在HTTP上触发(或者我应该使用Cloud Functions吗?)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！