基于Webrequest在气流上运行作业 [英] Running Job On Airflow Based On Webrequest

问题描述

我想知道是否可以通过HTTP请求获得气流任务。我对Airflow的调度部分不感兴趣。我只是想用它代替Celery。

I wanted to know if airflow tasks can be executed upon getting a request over HTTP. I am not interested in the scheduling part of Airflow. I just want to use it as a substitute for Celery.

因此，示例操作就是这样。

So an example operation would be something like this.

1. 用户提交请求某些报告的表格。


2. 后端接收请求，并向用户发送已接收到请求的通知。


3. 然后，后端使用Airflow计划作业以立即运行。


4. Airflow然后执行与DAG相关的一系列任务。例如，首先从redshift提取数据，从MySQL提取数据，对两个结果集进行一些操作，将它们组合在一起，然后将结果上传到Amazon S3，发送电子邮件。

1. User submits a form requesting for some report.
2. Backend receives the request and sends the user a notification that the request has been received.
3. The backend then schedules a job using Airflow to run immediately.
4. Airflow then executes a series of tasks associated with a DAG. For example, pull data from redshift first, pull data from MySQL, make some operations on the two result sets, combine them and then upload the results to Amazon S3, send an email.

根据我在网上阅读的内容，可以通过在命令行上执行 airflow ... 来运行气流作业。我想知道是否有可以执行相同操作的python API。

From whatever I read online, you can run airflow jobs by executing airflow ... on the command line. I was wondering if there is a python api which can execute the same thing.

谢谢。

推荐答案

Airflow REST API插件将为您提供帮助这里。按照说明安装插件后，您只需点击以下网址： http：// {HOST}：{PORT} /admin/rest_api/api/v1.0/trigger_dag？ dag_id = {dag_id}& run_id = {run_id}& conf = {url_encoded_json_parameters} ，将dag_id替换为dag的ID，忽略run_id或指定唯一的ID，并传递经过编码的url conf的json（使用触发的dag中需要的任何参数）。

The Airflow REST API Plugin would help you out here. Once you have followed the instructions for installing the plugin you would just need to hit the following url: http://{HOST}:{PORT}/admin/rest_api/api/v1.0/trigger_dag?dag_id={dag_id}&run_id={run_id}&conf={url_encoded_json_parameters}, replacing dag_id with the id of your dag, either omitting run_id or specify a unique id, and passing a url encoded json for conf (with any of the parameters you need in the triggered dag).

以下是使用jQuery调用Airflow api的JavaScript函数示例：

Here is an example JavaScript function that uses jQuery to call the Airflow api:

function triggerDag(dagId, dagParameters){
    var urlEncodedParameters = encodeURIComponent(dagParameters);
    var dagRunUrl = "http://airflow:8080/admin/rest_api/api/v1.0/trigger_dag?dag_id="+dagId+"&conf="+urlEncodedParameters;
    $.ajax({
        url: dagRunUrl,
        dataType: "json",
        success: function(msg) {
            console.log('Successfully started the dag');
        },
        error: function(e){
           console.log('Failed to start the dag');
        }
    });
}

这篇关于基于Webrequest在气流上运行作业的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！